Releasing a new version

We release a new version every four weeks on Wednesday. This lines up with our sprints, which are two weeks. Code freeze happens the Friday before, and "break the release" happens on Monday.

Because there are thirteen weeks in a quarter, there is always one extra out-of-sprint week at the end of each quarter. We dedicate that last week to cleanup tasks along with OKR reflection and planning.

This consistency is important, as it means our community and our customers can look forward to new features at a predictable pace. There will always be more work that we want to do and if we get in a habit of pushing deadlines out, we'll push them further and further.

Each release there is a different release owner in charge, according to this calendar.

If we've shipped features that we want to feature in the release notes, we use the label highlight on our pull request. If after the code freeze we have important bugfixes that we want to get into the release, we add the label release-[version]. This makes it easier for the release owner to figure out changes for the release blog post and to cherry-pick commits between the Code Freeze and the Release.

When we say "release", technically we mean releasing to self-hosted users as we deploy PostHog Cloud continuously. However, releases are still an important moment within PostHog as we publicly announce all new features.

Version numbers

Every release we bump the minor in major.minor.patch. At the moment, we're at version 1 for major. This will only change once we have released sufficient functionality under stage 2 of our Roadmap.

Hopefully we will not have to do many patch versions, but if between versions we discover a breaking bug, we will.

Timeline

💡 For the context of this guide [version] is interpreted as the version of the release (e.g. 1.29.0).

On the Friday before the release, we institute a code freeze. Feel free to make an announcement on Slack before we cut the branch, so people have a heads-up. Then, we branch master into release-[version] and deploy that to our playground environment, playground.posthog.com. We then host an hour-long "Break the release" session where everyone lends a hand in testing for any bugs. It's a recurring meeting, so you don't need to set it up.

Only bugfixes and finishing touches are allowed to be merged into this branch between the code freeze and the release going out. This gives us about three days to test the release.

The release manager is ultimately responsible for the timeline of the release. They are responsible for creating the "Code freeze" and "Break the release" calendar events as soon as possible. They should create these events under the Releases calendar linked up top.

Steps

Pre-release (Wednesday before the release)

1. Post in #dev about the upcoming release (replace <version> and <array draft pr> from Joe)

Release is happening next Wednesday. Which means 
1. There will be code freeze today (fixes that need to be cherry picked later should be tagged with `release-<version>`)
2. Please join the Break the Release meeting to help out testing on the Playground.
3. Shipped something awesome this month, please add a blurb or comment to <array draft pr> ([highlighted PRs](https://github.com/PostHog/posthog/pulls?q=is%3Apr+label%3A%22highlight+%3Astar%3A%22+)) :pray:

3. Start the release-[version] branch from master to initiate the code freeze.

4. Update the VERSION value in posthog/version.py and add an appropriate entry in posthog/versions.json. Then commit those changes:

   Terminal

   git checkout release-[version]
   git add posthog/version.py versions.json
   git commit -m "chore: Bump version to [version]"

5. Publish the release-[version] branch:

   Terminal

   git push -u origin release-[version]

   Note that this will result in a Docker image tagged release-[version]-unstable being built. It might take a while, but it should show up in Docker Hub within half an hour. You can check the build's status on the GitHub Actions page of the main repo.

   💡 Make sure you have doctl, helm, and k9s installed before going through the next steps. You can install all of these with brew install doctl helm k9s.

6. Upgrade PostHog playground

   1. The PostHog Playground uses a helm chart deployment on DigitalOcean. Find the playground cluster in our DigitalOcean Kubernetes clusters list.

   2. If this is your first time on DigitalOcean, you'll see the below screen. If it's not, or you don't see the Getting Started flow, click "Remind me how to use this file to connect to the cluster" in the "Config file" section under the "Overview" tab. Click Get Started.

      

   3. Copy the automatic connection script by clicking the copy icon.

      

   4. Open terminal and run the command you copied. This command will set the correct kubectl context for the playground environment. As a sanity check, run kubectl config current-context and make sure that the current context name has playground in it somewhere.

   5. Optional: Open another terminal window and run k9s. Use the arrow keys to scroll down to the PostHog clusters and keep an eye on this for the duration of the upgrade. k9s is a terminal GUI that makes it easier to manage and observe your deployed Kubernetes applications.

   6. Get the latest values and store them in a playground.yaml file:

      Terminal

      # use tail to remove the first line "USER SUPPLIED VALUES:"
      helm get values posthog -n posthog | tail -n +2 > playground.yaml

   7. Update the playground.yaml for image: -> tag: value to release-[version]-unstable with the new version.

   8. Follow the upgrade instructions here. Replace values.yaml in the last upgrade command with playground.yaml.

      ⚠️ Note that you might need to follow major upgrade notes as mentioned in the upgrade guide, the same way our users would be required to. If so, make any additional changes to the playground.yaml file as needed.

   9. Optional: Keep an eye on the progress of the upgrade in k9s

   10. If the helm upgrade command fails or if in the end the output for kubectl get pods -n posthog doesn't show everything as running, then ask team-platform for guidance.

       ⚠️ Make sure you're not in a working directory containing posthog folder, this could lead to the upgrade command looking for the chart locally rather than using the helm repo installed and seeing an error like Chart.yaml not found.

   11. Optional: Verify playground is running the latest image by running kubectl get pod --namespace posthog. In the output of that command, you should see a row like posthog-web-6447ff5fdf-gs664. Copy this row (the numbers after posthog-web- will be different), and then run kubectl describe pod --namespace posthog posthog-web-6447ff5fdf-gs664. If you scroll up in that output, you should see a line like Image: posthog/posthog@sha256:daf43a4a4cd06658e41273bb8fe4a74f17b295d67c6f1e16c17243b5d09af7ee. This is the sha of the image that is running. You can compare this to the sha in Docker Hub to verify that the image is the latest.

   12. Go to the playground and test that everything is working as expected. Check that the version running is the same as the one we're releasing.

   13. Commit the changes to the playground.yaml file in the vpc repo - have someone from Infrastructure team review.

7. Time for the "Break the release" session! It's imperative that the session uses the published release-[version]-unstable image from Docker Hub to avoid any potential bugs creeping up in the final build stage. You're responsible for running the session, prepare the release checklist doc by adding the template at the top. Note that you're also responsible for making sure everything is tested and for cherry picking the fixes and prs tagged with release-<version> into the release branch.

8. Figure out what's updated in this release with the command below or by asking the Product or Engineering Team. The command will output the entire commit list to changelog.txt, sorted by PR type and scope. You can use this list to obtain external contributions to highlight in the Array. In addition, you can look for the highlight tag in PRs but be mindful it's not used very consistently.

   Terminal

   git checkout release-[version]
   git log --pretty=format:"%s %ae" origin/release-[old-version]..head | sort -t ':' -k 1,1 -s > changelog.txt

9. Write up the PostHog Array blog post. Please tag Joe Martin for review, as this helps Marketing coordinate other announcements. Do not release the post until the day of release.

10. Share the PostHog Array blog post with all partners listed in the PostHog partner directory via the dedicated Slack channels. Don't have access to them all? Please ask Joe Martin to do this instead.

Launch (day of the release)

1. Tag the version in GitHub. This will also build and push the release-[version], latest-release (for both PostHog base & FOSS) Docker images to Docker Hub. Please do this once the release branch is finalized, some users may see the image on Docker Hub and update immediately.
   Terminal

   git tag -a [version] -m "Version [version]"
   git push --follow-tags

2. Create a new charts-clickhouse branch named bump-[version] to update the Helm chart:
   1. In Chart.yaml update appVersion to the new version.
   2. In Chart.yaml update version by increasing the minor version by 1.
   3. In values.yaml update image.default to point to the new tag (i.e. :release-[version]).
   4. In ALL_VALUES.md update the default value of image.default to what you set in the previous step. Also, update the AppVersion shields.io badge at the top.
   5. Push the relevant changes and create a PR. Do not merge until the release-[version] branch is built. (You can see that in Docker Hub)
3. Publish the PostHog Array blog post.
4. Create a new main repo (posthog) branch named sync-[version]. Cherry-pick the release-[version] commits updating version.py and versions.json into sync-[version] and create a PR to get them into master. Merging this to master will notify users that an update is available. The Array post should be out at this point so that the "Release notes" link isn't a 404.
5. Go to the EWXT9O7BVDC2O CloudFront distribution to the "Invalidations" tab and add a new one with /* value. This will refresh the CloudFront cache so that users can see the new version. You can check this by visiting https://update.posthog.com/
6. Send a message on the PostHog Users Slack (community) in #announcements to let everyone know the release has shipped.
7. Send the newsletter with the PostHog Array. The Marketing Team will arrange this, provided Joe Martin has been tagged for review in the PostHog Array blog post.