CI/CD

How every repository is checked, and how a service gets from a merge to production.

This document is the design: the pipeline, the contract each repository signs, and why. The operator's side - the release-day rule, the checklist, rollback, and hotfixes - is release.md. The developer's side - where to branch from, what a merge request must look like, and how a release is chosen when several people have work in flight - is branching.md.

One model for every repo: a merge request is checked, a merge to main ships to staging, and a version tag promotes the image staging already ran to production.

Why

A survey of the team's repositories on 2026-09-17 found no two pipelines alike:

Concern	What the repositories did
Checks	Blocking in the templates, crawler-service and go-sdk; security scanners allowed to fail in parser-service; nothing blocking in backend and hub; no checks at all in aioz-depin-wrapper and go-aioz-media
Migrations	A manual CLI job (backend, aioz-stream), `m.Up()` at startup (crawler-service, depin, hub), or by hand from a Makefile
Deploy	backend copies binaries over SSH and restarts containers; aioz-stream runs a blue/green script; nothing else deploys from CI
Branches	`develop`/`main`, `develop`/`stag`/`main` plus rc tags, `release-`, and `only:` lists naming feature branches

aioz-stream, the most complete pipeline, reaches the same staging environment three ways (develop*, stag, and an rc tag), deploys nothing from main, and needs two to three chained manual buttons per release. People could not tell from the pipeline page what would ship where.

Each of those has a cost:

A check that does not block is a suggestion. A red job with a green merge button teaches people to ignore red jobs.
Migrations at startup race. Every replica applies the schema on boot, and a failed migration is an outage instead of a failed pipeline.
A rebuild for production is a different image. Production should run the bytes staging ran, not bytes that ought to be identical.

The model

main is the only long-lived branch. Everything else is a short-lived branch merged into it by merge request. The branch rules themselves - naming, lifetime, rebasing, and the one case that justifies a support/ branch - are in branching.md.

merge request  -> lint, test, image:check                         blocking
               -> validate                                       warns, see Checks
merge to main  -> checks
               -> image:build        sha-<commit>, once
               -> migrate:staging    golang-migrate up
               -> deploy:staging     health check must pass
               -> mark:staged        tag the digest staged-sha-<commit>
tag vX.Y.Z     -> checks
               -> image:resolve      needs staged-sha-<commit>
               -> approve:prod       the one manual button
               -> migrate:prod       automatic after approval
               -> deploy:prod        automatic
rollback:staging, rollback:prod      manual, independent

Rules

One build per commit. image:build runs on main only and pushes sha-<commit>. Nothing else builds a deployable image.
Production only ships what staging ran. After a successful staging deploy the digest gets a second tag, staged-sha-<commit>. A release tag on a commit without it fails image:resolve and nothing reaches production.
One approval per release. approve:prod is the only button. Migrate and deploy follow it automatically, in that order.
A tag must point at a commit on main. vX.Y.Z tags and main are protected; only maintainers push them.
Migrate before deploy. A failed migration stops the pipeline with the old release still serving.
Rollback moves traffic, not schema. It needs no registry and no artifact, so it works when the registry or network is what broke.

Why a registry tag and not asking the staging host what it runs: staging moves forward on every merge, so it is usually already running something newer than the commit being tagged. The question that matters is whether this digest was deployed to staging and passed its health check, and the staged-sha- tag records exactly that. A tag on an existing digest adds no layers, which matters because the registry cannot delete.

Hotfixes

Merge the fix to main through a normal merge request. It reaches staging on merge. Tag it and approve. There is no release branch and no cherry-pick back. See release.md for when a hotfix is warranted and what to check before tagging one.

Profiles

Repositories include the shared jobs from the aioz-ci-template repository instead of copying them, pinned to a version:

yaml

include:
  - project: tuan.quang.tran/aioz-ci-template
    ref: v1.0.0
    file: profiles/library.yml

That is the whole file for a repository that needs nothing else. A service will additionally declare, once the service profile exists:

yaml

variables:
  SERVICE_IMAGES: "api=deploy/docker/api.Dockerfile"
  MIGRATIONS_DIR: internal/migrations

Profile	For	Jobs
`library`	Go modules with no image: aioz-common, aioz-config, go-sdk	lint, validate, test
`service`	Anything deployed as a container	library jobs, plus image, migrate, deploy, rollback

Binaries released over FTP (hub, aioz-depin-wrapper, go-aioz-media) use the library profile for their checks. Their release flow is out of scope for v1.

Repo-specific jobs (aioz-stream's HAProxy config check, SDK publishing) stay in the repo's own .gitlab-ci.yml and extends: the shared hidden jobs.

The library profile exists as of v1.0.0. The service profile does not yet: a repository needing image, migrate and deploy keeps those jobs in its own .gitlab-ci.yml for now and includes the library profile for its checks.

What a repository provides

Every repository

Item	Contract
`.golangci.yml`	v2 schema (`version: "2"`)
`.gitleaksignore`, `.trivyignore.yaml`	Every entry has a reason; trivy entries have an expiry

The shared jobs call the tools directly, so no Makefile is required: a repository with none at all - aioz-logger has none - can adopt the library profile unchanged. An earlier draft of this document required make lint-verify, make test and make test-build; no library repository had the first or the third, and a template that demands them onboards nobody.

Those targets are still worth having, as the local reproduction of what CI does:

Target	Reproduces
`make lint`	`golangci-lint run ./...`
`make lint-verify`	The config check the lint job runs first
`make test`	`go test -race ./...`

Two optional variables hand a repository's own extra checks to the shared test job, so it does not have to override the job and lose the rest of it:

Variable	Runs	Example
`GENERATE_VERIFY_CMD`	First, before the tests	`make swagger-verify` in aioz-stream
`TEST_BUILD_CMD`	Second, before the tests	`make test-build`

Unset means skipped, and the job log says so.

Services, in addition

Item	Contract
Dockerfiles	Listed in `SERVICE_IMAGES` as `name=path`, images on the approved list
Migrations	golang-migrate files in `MIGRATIONS_DIR`, copied into the application image with the golang-migrate CLI at the version in `go.mod`. Applied only by `deploy.sh migrate`, never by the app at startup (why)
`deploy/deploy.sh`	Meets the deploy script contract below. Shipped to the host by every migrate and deploy job, with `docker-compose.deploy.yml` and `.env.images`
CI/CD variables	`DEPLOY_HOST`, `DEPLOY_PATH`, `DEPLOY_SSH_PRIVATE_KEY` (base64) and `DEPLOY_SSH_USER`, each scoped to the `staging` and `production` environments; protected and masked
Host files	`.env`, `.env.migrate` (`DATABASE_URL` of the migrator role) and the service's config. Kept on the host, never shipped by CI
`.env.images`	One `<NAME>_IMAGE=<repo>@sha256:<digest>` line per image in `SERVICE_IMAGES`, and nothing else: no secrets. Written by CI on every migrate and deploy job, and by `deploy.sh rollback`; it always names the images that run, or are about to

Deploy script contract

Every service has a deploy/deploy.sh, and every one of them answers the same commands with the same meaning. How a script does it is the service's business: blue/green, stop-then-start, or a plain recreate, whatever its roles need. What it answers is not. The point is that anyone, a person at 02:00 or a CI job, can migrate, deploy, restart, roll back or inspect any service the same way, without reading its script and without changing any code. An operation that needs an edit to the script, the compose file or the pipeline is a gap in the script, not a step in a runbook.

Command	Required	Must
`deploy.sh migrate`	When the service owns a database	Apply pending migrations as a one-off container from the image `docker-compose.deploy.yml` gives the migrate service, read from `.env.images`, then exit. Refuse a dirty schema and print the `force` command for a person. Never start, stop or restart a running role
`deploy.sh deploy [ROLE ...]`	Always	Pull every image the roles name in `.env.images` before changing anything, then put each role whose running image differs onto it. No role means every role. Wait for the health check (and readiness, for a role behind HAProxy) before moving traffic or reporting success
`deploy.sh deploy all`	Always	Deploy everything in `docker-compose.deploy.yml`, so one command brings up a fresh host: start every data store and HAProxy that is not running and wait for its health check, then deploy every role as above. A data store or HAProxy that is already running is left as it is, never recreated
`deploy.sh redeploy [ROLE ... \| all]`	Always	Restart roles on the image they already run, with the same health gates as `deploy`. No pull, and `.env.images` is not read. No role, or `all`, means every role. For a changed host config or a stuck process
`deploy.sh rollback [ROLE ... \| all]`	Always	Put roles back on the image they ran before, from what is already on the host, without the registry, and write those images back to `.env.images` so the file matches what runs. No role, or `all`, means every role. Never un-migrate
`deploy.sh status`	Always	Print one line for every service in `docker-compose.deploy.yml` - roles, data stores, HAProxy - as `<service> <state> <image>`, with the image last. Then the schema version, when the service has one

Every command:

runs on the host from the compose directory, with nothing installed beyond docker;
takes no image on the command line: images come only from .env.images, because roles of one service may run different images;
refuses an image in .env.images that is not a digest (<repo>@sha256:...), so staging and production provably run the same bytes;
exits non-zero on any failure, leaving whatever was serving before still serving where it can;
logs one JSON object per line on stdout, with a deploy_id per run, and records every change it makes in a history file on the host;
echoes each docker command that changes something to stderr, so a run can be taken over by hand halfway through.

all is a reserved word: no role may be named all. A script may add commands of its own (bootstrap, reconcile, logs); it may not change the meaning of these five.

Migrate and deploy are separate CI jobs in separate stages: migrate:<env> in a migrate stage, then deploy:<env> in a deploy stage that needs it, both holding resource_group: <env>. Each writes the release's digests to .env.images on the host, replacing the file in one step (write a temporary file, then rename it), and then calls only the commands above. docker-compose.deploy.yml references them as ${API_IMAGE}, ${WORKER_IMAGE} and so on, and the script runs compose with both --env-file .env --env-file .env.images.

Migrations

Every migration must work with the release before it. Migrate runs while the old release is still serving, so for a window the previous binary runs against the new schema.

Change	Safe in one release?
Add a table, a nullable column, or a column with a default	Yes
Add an index (`CONCURRENTLY` on large tables)	Yes
Drop or rename a column or table	No: stop using it in release N, drop it in N+1
Add `NOT NULL` without a default	No: add nullable and backfill in N, constrain in N+1

File names are timestamps, never sequence numbers

A migration's version is its filename prefix, and it MUST be a UTC timestamp:

20260924103000_add_invoice_export.up.sql
20260924103000_add_invoice_export.down.sql

Create them with the tool, not by hand, and do not pass -seq:

migrate create -ext sql -dir $MIGRATIONS_DIR -format 20060102150405 add_invoice_export

Sequence numbers (000007_...) fail in two different ways, and both of them land on main rather than in review.

Two developers pick the same number. Both branches are green, because each one only sees its own file. Git merges them without a conflict, because they are different filenames. Then migrate refuses the directory outright with a duplicate version error, and the first person to notice is whoever merged second, on staging, at the point where the deploy stops.

The worse one: a lower version is silently skipped. golang-migrate keeps a single current version in schema_migrations and up applies only versions greater than it. A branch that was parked for three weeks carries a number, or a timestamp, from before the migrations that have since gone out. Nothing errors. The migrate job is green, the deploy is green, and the table simply does not exist. The failure surfaces later as a runtime error on a code path nobody may touch for days.

Timestamps make the first failure nearly impossible, since a collision needs two migrate create calls in the same second. They do not fix the second one: a branch's timestamp ages along with the branch, and it is already too low the moment someone else's newer migration merges first.

Re-stamp migrations before every merge to `main`

Before merging any branch that adds migrations, give them fresh versions. A parked branch needs it most, but a two-day branch needs it too if another migration reached main in the meantime, and you cannot tell that from your own branch. Use templates/migrations/restamp.sh:

BASE=origin/main MIGRATIONS_DIR=internal/migrations ./restamp.sh --dry-run
BASE=origin/main MIGRATIONS_DIR=internal/migrations ./restamp.sh

It renames with git mv, so the rename is staged and reviewable. Four details matter, and all four are why this is a script and not a one-line for loop:

Detail	Why
Rename `.up.sql` and `.down.sql` to the same new version	They are one migration. A `date` call per file can straddle a second boundary and split them.
Keep the branch's own migrations in their original order, one second apart	Two migrations on one branch usually depend on each other. Collapsing them to one timestamp makes the order undefined.
Refuse to run when the clock is behind the newest version on `main`	Renaming to a version that is still too low reproduces the exact bug you are fixing, silently.
Compare versions in base 10	A legacy sequential name such as `000008` is not valid octal, and the shell aborts on it.

Do it as the last step before merging, after the final rebase, in a commit of its own. Then reset your local database and re-run the tests: your local schema_migrations still records the old versions, so an up against it skips the files you just renamed, which is the same failure in miniature.

Never re-stamp a migration that has already been applied anywhere shared. Staging and production keep the old version in schema_migrations, so the renamed file counts as a new migration and runs a second time. This is only ever for migrations that have not left the branch.

This is a static, checkable property, so migrate:lint should fail any merge request whose new migration version is lower than the highest version already on main. That rule does not exist in the job yet; see release.md.

A dirty schema stops the migrate job. Someone looks at it, fixes it by hand, and runs migrate force <version>; the pipeline never guesses.

Rollback does not un-migrate. That is why every migration must be backward compatible: rolling back the binary must be enough. migrate:lint and compat:check enforce that; see release.md.

Checks

Lint and test block the merge. Set Settings -> Merge requests -> Pipelines must succeed on every project; without it a red pipeline is only a red icon.

Stage	Jobs	Blocking?
lint	golangci-lint v2, findings shown inline on the MR	yes
validate	govulncheck, gitleaks, trivy-fs, hadolint (see `templates/ci-security` for how to triage)	no, warns for now
test	`GENERATE_VERIFY_CMD` and `TEST_BUILD_CMD` if set, then `go test -race ./...`	yes
image:check	Services only: build every image without pushing, so a broken Dockerfile fails the MR and not `main`	yes

Why validate warns

The section above argues that a check which does not block is a suggestion, and that still holds. The validate stage is the deliberate exception, for now.

The older libraries carry a backlog of findings nobody has triaged, and govulncheck reports transitive vulnerabilities that cannot be fixed from the repository reporting them - it has no ignore file, so such a finding cannot even be suppressed. Blocking today would leave those pipelines permanently red, which teaches people to ignore the whole stage: the exact failure this standard exists to remove. The stage still fails, as a yellow warning on the pipeline and in the merge request, so a new finding is visible in every merge request.

A repository that has cleared its findings enforces them one job at a time, in its own .gitlab-ci.yml:

yaml

validate:gitleaks:
  allow_failure: false

That is a per-job override rather than the VALIDATE_ALLOW_FAILURE variable an earlier draft of this document specified: GitLab requires allow_failure to be a boolean literal, so a variable-driven switch means duplicating every ref rule into a blocking and a warning variant.

Flipping the default - dropping allow_failure from .aioz-validate in the template repository - waits on the libraries being clean. Track it per repository rather than doing it all at once.

Adding CI to a repository

The template is released and in use. Adopting it is one merge request, and most of the work is in step 1.

1. Get the checks green locally, first

Turning on a blocking job over a backlog of findings teaches people to ignore the job. Run what CI will run, before CI runs it:

go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2 run ./...
go test -race ./...

and the four scanners, whose commands and triage guidance are in templates/ci-security. Fix what is real; write the rest into .gitleaksignore and .trivyignore.yaml with a reason per entry and an expiry on the trivy ones.

2. Add the repository files

File	Needed	Note
`.golangci.yml`	Yes	v2 schema - the first line is `version: "2"`. A v2 config in a v1 binary fails with `the format is required`
`.gitleaksignore`	If gitleaks found anything	One reason per entry
`.trivyignore.yaml`	If trivy found anything	Reason and expiry per entry

No Makefile is required. The shared jobs call the tools directly, so a repository with no Makefile at all adopts the profile unchanged.

3. Add the include

This is the entire .gitlab-ci.yml for a library:

yaml

include:
  - project: tuan.quang.tran/aioz-ci-template
    ref: v1.0.0
    file: profiles/library.yml

Pin an exact tag. ref: main is not supported: a change to the template would otherwise reach every repository unreviewed, which is the copy-paste problem in a new shape.

Name a profile, never a jobs/ file. The jobs/ layout is internal and may be rearranged within a major version.

4. Private modules, if the repository imports any

Repositories importing from gitlab.internal or 10.0.0.50 opt in per job - most repositories need nothing here:

yaml

lint:
  before_script:
    - !reference [.aioz-go-private, before_script]
    - !reference [.aioz-lint, before_script]

test:
  before_script:
    - !reference [.aioz-go-private, before_script]
    - !reference [.aioz-test, before_script]

Order matters: the credentials must be in place before the job's own go list fetch runs.

Then, in the project that owns the private module: Settings -> CI/CD -> Job token permissions -> Allowlist, and add the consuming project. Without it the clone returns 403 with a message that does not mention the allowlist, which costs an afternoon the first time.

5. Turn on the project settings

Two settings, and without them the pipeline is decoration:

Settings -> Merge requests -> Pipelines must succeed. Otherwise a red pipeline is a red icon next to a live merge button.
Settings -> Repository -> Protected branches and tags: protect main and v*.

6. Open the merge request

Read the pipeline rather than glancing at the colour. lint and test block; the four validate:* jobs warn, so a yellow pipeline still merges and still means someone should look.

Four things that catch people out

stages: replaces, it does not merge. A repository adding a stage of its own must redeclare the shared three as well, or the shared jobs land in an undeclared stage and the pipeline fails to create:
yaml
```
stages: [lint, validate, test, integration]
```
rules: replaces too. A job that extends .aioz-test or .aioz-lint without also extending .aioz-rules inherits no rules, so it runs on every branch push. A repository needing extra refs must list every ref it wants, including merge requests, the default branch and tags.
No YAML anchors. An anchor is resolved by the YAML parser before GitLab merges files, so an alias cannot see an anchor defined in an included file - the key silently goes missing. Use extends: and !reference, which resolve after the merge. The template's own pipeline fails if an anchor appears.
Repo-specific jobs stay in the repo. aioz-stream's HAProxy config check, SDK publishing, an integration suite - keep them in the repository's own .gitlab-ci.yml and extends: a shared hidden job.

Services

The service profile does not exist yet, so a service includes the library profile for its checks and keeps image, migrate, deploy and rollback in its own .gitlab-ci.yml for now. It also needs the CI/CD variables and deploy/deploy.sh described in what a repository provides, then:

Move any startup migration into MIGRATIONS_DIR and delete the code that ran it. m.Up() at boot races across replicas and turns a bad migration into an outage instead of a failed pipeline.
Make deploy/deploy.sh meet the deploy script contract, and add migrate:<env> and deploy:<env> as separate jobs.
Merge, watch staging migrate and deploy, then tag and approve the first release - release.md.

Reference

The full variable list, the hidden jobs, the versioning policy and how to develop the template are in aioz-ci-template's own README. This section is the procedure; that is the reference.

Order

Pilots, in order: aioz-common (library, piloted first), then the remaining libraries - aioz-config, aioz-logger, aioz-stats, go-sdk - then parser-service and aioz-stream once the service profile lands.

Scope

Covered: Go repositories on GitLab, the shared linux runner, the registry at 10.0.0.106:5000, staging and production environments.

Not yet covered: multi-platform binaries released to FTP, Node repositories, Kubernetes, and a develop environment separate from staging.

CI/CD ​

Why ​

The model ​

Rules ​

Hotfixes ​

Profiles ​

What a repository provides ​

Every repository ​

Services, in addition ​

Deploy script contract ​

Migrations ​

File names are timestamps, never sequence numbers ​

Re-stamp migrations before every merge to main ​

Checks ​

Why validate warns ​

Adding CI to a repository ​

1. Get the checks green locally, first ​

2. Add the repository files ​

3. Add the include ​

4. Private modules, if the repository imports any ​

5. Turn on the project settings ​

6. Open the merge request ​

Four things that catch people out ​

Services ​

Reference ​

Order ​

Scope ​

CI/CD

Why

The model

Rules

Hotfixes

Profiles

What a repository provides

Every repository

Services, in addition

Deploy script contract

Migrations

File names are timestamps, never sequence numbers

Re-stamp migrations before every merge to `main`

Checks

Why validate warns

Adding CI to a repository

1. Get the checks green locally, first

2. Add the repository files

3. Add the include

4. Private modules, if the repository imports any

5. Turn on the project settings

6. Open the merge request

Four things that catch people out

Services

Reference

Order

Scope