Breaking glass to debug PostHog Cloud

We've all been there. Something was just merged and now there is a bug that you are having a real hard time pinning down. You hate to do it...but you need to get on a prod box to see what's going on. SHAME

Step 1

Make sure that you have awscli installed locally on your computer.

For macOS you should brew install it:

brew install awscli

Configure AWS CLI

aws configure

From here follow the wizard and enter your AWS Key ID and Secret Key. You should default to us-east-1 as your region.

Step 2

You'll need to make sure that you have the Session Manager plugin for AWS ClI installed. Follow the steps below (for macOS) to install the Session Manager plugin using the bundled installer.

Download the bundled installer.

cd ~/awstemp
curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/mac/sessionmanager-bundle.zip" -o "sessionmanager-bundle.zip"

Unzip the package.

unzip sessionmanager-bundle.zip

Run the install command.

sudo ./sessionmanager-bundle/install -i /usr/local/sessionmanagerplugin -b /usr/local/bin/session-manager-plugin
rm -r ~/awstemp

Step 3

Go to the ECS console in AWS. Select the posthog-production-cluster and then select the service that you would like to exec into (most likely a posthog-production or worker task).

From that service select a currently running task. IT MUST BE RUNNING AND STABLE. If the service is flapping this will not help you.

Copy the TASK ID (should look something like this 5627220248b6289d11ba5465524902b9). We'll be using that later.

Step 4

Exec into the container

Plug the Task ID (from the previous step) into the following command and get to work slacker! (You may also need to change the container name depending on the service you are hoping into)

aws ecs execute-command  \
    --region us-east-1 \
    --cluster posthog-production-cluster \
    --container posthog-production \
    --command "/bin/bash" \
    --task <TASK_ID from earlier> \
    --interactive

If you need a Django shell, just run the following after connecting

python manage.py shell_plus

If all of this fails reach out to the engineer on call.