Team Playbook

Appearance

Releasing ​

How a change gets from main to production, how it comes back if it is wrong, and what to do when it cannot wait.

A release is always two steps in one order: migrate, then deploy. The pipeline does both after a single approval. This document is the operator's side of the model described in ci-cd.md - what you press, what you check, and what to do when a step fails. How the commit you are tagging got onto main in the first place is branching.md.

Release days ​

We release on Monday and Wednesday. Nothing else ships to production except a hotfix.

Tag and approveMonday or Wednesday, 10:00-15:00
Cutoff for the mergeEnd of the previous working day. A commit merged the morning of a release day is not released that day
No releasesThursday, Friday, weekends, and the day before a public holiday

Why these two days:

  • A commit needs a night on staging. The cutoff exists so that every release candidate has run through at least one full day of verification and one nightly job cycle before it is tagged.
  • Monday gives the week a known start. Work merged late the previous week ships first thing rather than sitting.
  • Wednesday leaves two working days of cover. Whoever released is still at a desk on Thursday and Friday when a slow-burning problem shows up. A Friday release is discovered on Saturday by whoever is on call, with the author unreachable, which is how a small bug becomes an incident.

A release outside these days needs the lead's agreement and is recorded as a hotfix, with the reason it could not wait.

Before you tag ​

The person releasing checks all of these. If any is false, the release waits.

  • The commit is on main, and its work was verified and accepted on that build version on staging. The version is in the app footer or /version; the format is #{pipeline_id}-{short_commit_ref}.
  • Every work item in the Jira version vX.Y.Z is in Ready for Release, and its verified build version is at or below the commit you tag.
  • Staging has been running that commit for at least one working day.
  • migrate:staging succeeded on that commit, and compat:check is green (see N-1 compatibility).
  • Every migration in this release is backward compatible with the release currently in production.
  • Anything not ready for users is behind a feature flag that is off.
  • You know what the previous production tag is. You will need it for the rollback decision.

Releasing ​

  1. Pick the commit. The one whose work was verified and accepted. Its build version names it.

  2. Tag it.

    git tag v1.4.0 <commit>
git push origin v1.4.0

    Version numbers follow semver: breaking API change bumps major, new behaviour bumps minor, fix-only bumps patch.

  3. Wait for image:resolve. It checks the registry for staged-sha-<commit>, the tag written when that exact digest passed its staging health check. If the tag is missing, the commit never deployed to staging cleanly and the pipeline stops here. Do not work around it.

  4. Announce. Post in the team channel: version, what is in it, and one line on whether it carries a migration. This is what someone reads at 02:00 when a graph turns red.

  5. Press approve:prod. The lead approves. This is the only button in the release.

  6. migrate:prod runs. golang-migrate applies every pending migration while the old release is still serving traffic. If it fails, the pipeline stops and production is untouched and still healthy. Go to a failed migration.

  7. deploy:prod runs. The new image starts in the idle slot, its health check must pass, and only then does HAProxy move traffic. See zero-downtime deploy. If the health check fails, the job exits non-zero and traffic never moves - the old release is still serving.

  8. Watch for 30 minutes. Error rate, p99 latency, and the saturation of whatever the change touched. Keep the terminal open; you are the person who presses rollback.

  9. Confirm. Post the build version and "healthy" in the channel.

  10. Release the Jira version. In the repository's Jira project, release version vX.Y.Z. That moves every work item in it from Ready for Release to Done (jira.md). A work item that missed the release keeps its status; move it to the next version.

Migrate before deploy ​

The order is not arbitrary, and it is the reason every migration must be backward compatible.

t0   schema N-1   binary N-1        steady state
t1   schema N     binary N-1        migrate:prod finished, deploy has not started
t2   schema N     binary N          deploy:prod finished

At t1 the old binary is running against the new schema, for anywhere between seconds and minutes. That window is real, and it is also exactly the state a rollback returns you to. So:

The schema at version N must serve binary N-1. Always.

The alternative - deploy first, then migrate - puts the new binary in front of the old schema, which fails on the first request that touches a new column. There is no ordering that avoids a mixed state, so we pick the one where the mixed state is survivable and where a failure leaves the old release intact.

Migrate is its own process and its own job ​

The application never migrates. No role runs m.Up(), AutoMigrate or migrate up at startup, and no entrypoint or CMD wraps a migration before the binary. Migrating at boot breaks every guarantee above:

  • Replicas race. Every container that starts applies the schema, two at once on a blue/green switch.
  • A bad migration is an outage, not a failed job. The new container crash-loops while the pipeline is already past the point where it could stop.
  • The order is lost. The schema changes when the new binary starts, so there is no t1 where the old release serves the new schema, and no point where a failure leaves the old release untouched.
  • Rollback is unsafe. Restarting the previous image can run its own, older migrations against a schema that has moved on.

Instead:

RuleHow
A one-off container, once per releasedeploy.sh migrate (contract), with the release's digests already in .env.images, runs the migration in a container of its own, for example a compose migrate service behind profiles: [migrate] started with docker compose run --rm, then exits. It touches no running container.
The same image as the deployThe golang-migrate CLI and the .sql files are built into the application image, so the schema applied is the one that was built and ran on staging.
Its own credentials.env.migrate on the host holds DATABASE_URL for a migrator role, the only role with DDL rights. The application roles connect with DML only, so a bug in the app cannot change the schema.
Its own CI job, in its own stagemigrate:staging and migrate:prod run in the migrate stage, before the deploy stage. deploy:<env> needs the migrate job, and both hold resource_group: <env>, so a failed or dirty migration stops the pipeline with the previous release still serving.
The app only checks/health reports not-ready while the schema is behind the version the binary expects. It never applies it.

Every deploy.sh command, migrate included, is defined once in the deploy script contract.

What is safe in one release ​

ChangeSafe in one release?
Add a table, a nullable column, or a column with a defaultYes
Add an index (CONCURRENTLY on large tables)Yes
Widen a type (varchar(50) -> text, int -> bigint)Yes
Add a CHECK or FOREIGN KEY as NOT VALID, validate laterYes
Drop or rename a column or tableNo. Stop using it in release N, drop it in N+1
Add NOT NULL without a defaultNo. Add nullable and backfill in N, constrain in N+1
Add a unique constraint to an existing columnNo. Build the index CONCURRENTLY in N, attach the constraint in N+1

The pattern for everything in the "No" half is expand, migrate, contract:

  1. Release N - expand. Add the new shape. Write to both old and new. Read from the old.
  2. Release N+1 - migrate. Backfill. Read from the new. Still write both.
  3. Release N+2 - contract. Stop writing the old. Drop it.

Three releases is a week and a half on a Monday/Wednesday cadence. Plan the column rename with that in mind rather than trying to do it in one go.

N-1 compatibility ​

"Every migration must be backward compatible" is only a rule if something checks it. Two things do.

1. The static check - migrate:lint ​

Runs on every merge request. It scans the diff under MIGRATIONS_DIR for statements that cannot be N-1 safe on their own:

DROP TABLE        DROP COLUMN        RENAME
SET NOT NULL      DROP DEFAULT       ALTER COLUMN ... TYPE (narrowing)
ADD CONSTRAINT ... UNIQUE            ADD CONSTRAINT ... CHECK (without NOT VALID)

A hit fails the job. To pass, the migration file carries an explicit marker on its own line stating which earlier release removed the last reader or writer:

sql
-- n-1-safe: users.legacy_email stopped being read in v1.3.0 and written in v1.3.2
ALTER TABLE users DROP COLUMN legacy_email;

The marker is a claim by a person, not a proof. Its value is that it is written down, is in the diff, and a reviewer has to agree with it. A reviewer who cannot find the release that stopped using the column rejects the MR.

The job also checks version ordering: a new migration whose version is lower than the highest version already on main fails. up applies only versions above the one in schema_migrations, so a stale file is skipped in silence rather than rejected, and the missing table shows up as a runtime error days later. The fix is to re-stamp the file; see file names are timestamps.

2. The behavioural check - compat:check ​

Runs on main after migrate:staging, and it is the one the release checklist asks about. It answers the question the static check cannot: does last release's binary still work against this release's schema?

compat:check
  1. start a scratch Postgres, restore the anonymised staging snapshot
  2. apply every migration up to HEAD                 -> schema N
  3. pull the image for the current production tag    -> binary N-1
  4. boot it against the scratch database
  5. its /health must pass, then run `make test-smoke` against it

Step 5 is what makes this worth having, so the smoke suite has to be real: it covers a write on every table this release's migrations touch, not just a SELECT 1. A migration that adds NOT NULL to a column the old binary leaves empty passes steps 1-4 and fails step 5, which is the entire point.

compat:check is advisory on a repository's first two weeks on the standard and blocking after that. It does not run on the tag pipeline - by then the answer would arrive too late to be useful.

What neither check catches ​

  • Application-level meaning. A column that keeps its type but changes what the values mean passes both checks and breaks the old binary anyway. This is a review question.
  • Data volume. A migration that is correct and takes 40 minutes on the production table is still an outage, because migrate:prod holds the pipeline and the lock. Check the row count, and use CONCURRENTLY and batched backfills for anything over a million rows. A backfill belongs in a job, not in a migration.
  • Long-running transactions. ALTER TABLE waits for a lock and everything behind it queues. Set lock_timeout in the migration and let it fail fast rather than stall the site.

Zero-downtime deploy ​

This section is the api role. Roles with no HTTP traffic are below.

deploy.sh deploy never restarts the running container. It starts a second one and moves traffic with HAProxy, which is what makes both the deploy and the rollback instant and reversible.

The two slots ​

Each service runs in one of two slots, blue and green, two compose services (api-blue, api-green) that HAProxy reaches by name. Exactly one is live. HAProxy holds both as backend servers and decides which receives traffic:

haproxy
backend app
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    default-server check inter 2s fall 3 rise 2

    server blue  api-blue:8080  weight 100
    server green api-green:8080 weight 100 disabled

The runtime API socket is what the deploy script talks to. No config file is rewritten and HAProxy is never reloaded, so no connection is dropped:

haproxy
global
    stats socket /var/run/haproxy/admin.sock mode 660 level admin expose-fd listeners

The full config, the image and its compose service are on the HAProxy page.

The sequence ​

deploy.sh deploy api, with the new digest in .env.images, does this, and exits non-zero at any step that fails:

  1. Read the live slot. show stat over the socket. Say it is blue. The target is green.
  2. Start green on the new digest, with the new config and secrets.
  3. Wait for its health check - poll green's /health directly, not through HAProxy, until it returns 200 or a timeout expires. A failure here stops the deploy with blue still live and untouched. This is the step that makes a broken release a failed pipeline instead of an outage.
  4. Enable green. set server app/green state ready. Both slots now serve. HAProxy's own checks have to pass twice (rise 2) before it sends real traffic.
  5. Drain blue. set server app/blue state drain. Existing connections finish; no new ones arrive.
  6. Wait for blue to empty. Poll show stat until its scur is 0, up to the drain timeout. Long-lived connections (websockets, streaming) will not drain on their own - they get the timeout, then get cut.
  7. Stop blue, but leave its image on the host. That image is the rollback.

Two things make this work, and they are the service's responsibility, not the script's:

  • /health means ready, not alive. It returns 200 only once the database pool is connected, migrations are at the expected version, and caches are warm enough to serve. A health check that returns 200 the moment the process starts turns step 3 into a no-op and step 5 into an outage.
  • Shutdown is graceful. On SIGTERM the process stops accepting new requests, finishes what it is holding, and exits. Otherwise step 6 drains cleanly and step 7 kills requests anyway.

Anything that is not an HTTP service ​

Workers and consumers have no traffic to switch, so the slot dance does not apply. They rely on the drain state instead - on SIGTERM a role stops accepting new work and lets what is in flight finish, rather than cancelling it. The rules are in source-structure.md; the deploy consequences are:

  • consumer overlaps: start the new container, then SIGTERM the old and let it drain. The broker holds anything neither is ready for.
  • core cannot overlap. There must be exactly one, and its flock will refuse a second, so the old container drains and exits before the new one starts. The gap is the drain.

The role's stop_grace_period in docker-compose.deploy.yml, which is how long deploy.sh lets a stop take, must be longer than its shutdown.drain_timeout. Otherwise the deploy kills the container mid-drain and the drain bought nothing.

Rollback ​

rollback:prod is a manual job on the release pipeline, and it does exactly one thing:

set server app/blue state ready
set server app/green state drain

Traffic returns to the previous slot, which is still on the host, already healthy. It needs no registry, no build, and no network beyond the host, so it works when the registry or the build system is what broke. It takes seconds.

It does not un-migrate. The schema stays at N. That is the whole reason for the N-1 rule: rolling back the binary has to be enough on its own.

When to press it ​

Press it. The bar for rolling back is deliberately low - it is cheap, and the release will still be there in two hours.

SymptomAction
Error rate or p99 clearly worse than before the deployRoll back now, diagnose after
A new 500 that traces to this releaseRoll back now
Feature is broken but nothing else is degradedTurn the feature flag off. That is faster and narrower than a rollback
Something is wrong and you are not sure it is the releaseRoll back. Ruling the release out is worth more than being right
The migration is what is wrongRoll back the binary, then forward-fix

Announce it in the channel, and open an incident if users were affected - Incidents & Postmortems.

After a rollback ​

The tag stays. Do not delete it or move it; it records what was attempted. Fix forward on main, let it go to staging, and release the fix as the next patch version on the next release day - or as a hotfix if production is still degraded.

When the migration is the problem ​

Rolling the schema back is the last thing to reach for, not the first. A down migration against live production data drops whatever was written since the up ran, and the down file has rarely been tested at all.

In order:

  1. Roll the binary back. If the schema change is additive - which it is, if the rules above were followed - the old binary ignores it and the site is fine. Stop here. You now have time.
  2. Write a forward migration. A new numbered migration that corrects the problem: drops the index that is locking, widens the column back, removes the constraint. Ship it as a patch release through the normal pipeline.
  3. Restore from backup only for actual data loss or corruption, and only as an incident with the lead involved. It is not a release step.

A failed migration ​

migrate:prod failed, so deploy:prod never started and production is still serving the old release, healthy. There is no user impact yet. Do not retry the button.

golang-migrate marks the schema dirty when a migration fails partway. The next run refuses to touch it, on purpose. To recover:

  1. Read the job log for the statement that failed.
  2. Connect and look at the actual schema. Decide whether the failed migration applied fully, partially, or not at all.
  3. Finish or undo the partial work by hand, so the schema matches exactly one version.
  4. migrate force <version> to that version, clearing the dirty flag.
  5. Re-run the pipeline only once someone else has seen steps 2-4.

The pipeline never guesses at this and neither should you at 17:00 on a Wednesday. If it is not obvious, stop - production is fine - and pick it up with a second pair of eyes.

Hotfixes ​

A hotfix is an urgent production fix that cannot wait for Monday or Wednesday. It is a smaller release, not a different one: same branch, same pipeline, same approval. There is no release branch and no cherry-pick.

It qualifies if users are affected now: an outage, data being corrupted, a security hole, or a broken critical path with no workaround. Everything else - including something embarrassing that works - waits for the next release day.

  1. Roll back first if you can. If the bad change is in the release that is live, rollback:prod beats any hotfix, and it is already available. Only write a hotfix when there is nothing to roll back to, or when the bug predates the current release.
  2. Branch from main: hotfix/<short-description>.
  3. Keep it to the fix. No refactoring, no drive-by cleanups, no dependency bumps. The diff should be reviewable in two minutes, because it is going to be reviewed in two minutes.
  4. Avoid a migration. If the fix seems to need a schema change, it is probably not a hotfix. If it genuinely does, the N-1 rule still holds, with no compat:check to catch you, so a second person reads the migration.
  5. Open a PR into main. One reviewer approval, full pipeline green. The checks are not skipped - a hotfix is when a broken build hurts most.
  6. Merge. It deploys to staging automatically. Verify the fix there. Ten minutes of verification is cheaper than a second incident.
  7. Tag the merge commit with a patch bump (v1.4.1), approve, watch.
  8. Write down what happened. If it was worth waking someone for, it is worth a postmortem - Incidents & Postmortems.

If main holds unreleased work that must not ship alongside the fix, it should have been behind a flag. Turn the flag off and ship main as it is. Do not branch around main to avoid it; that is how you get a release branch and then two of them.

Who does what ​

Releaser (rotates weekly)Runs the checklist, tags, announces, watches for 30 minutes, presses rollback
LeadPresses approve:prod. Agrees to any release outside Monday/Wednesday
Work item ownerVerifies the work on a staging build version and posts the results
BAAccepts the work on the same build and sets the Fix Version
AuthorAvailable on release day if their change is in it. Writes the forward fix if it is not

Rolling back is the releaser's call and needs nobody's permission.