Source structure

Start here if you have just joined. This is what one of our Go backends looks like from the outside in: the two repositories you will be working across, what runs where, and how a request and a background job each find their way through the code. The rules further down are the same thing said precisely, for when you need to settle an argument.

The shortest version

Every backend service in the team is one Go module, one Docker image, one binary. That binary does several different jobs depending on which subcommand you give it - serve HTTP, run scheduled jobs, consume a queue, migrate the database. We call those roles.

Inside, the code is grouped by feature, not by technical layer. All of billing lives in one directory; all of user accounts lives in another. There is no controllers/ directory holding thirty unrelated controllers.

Anything genuinely shared between services - the logger, the config loader, the database pool, the retry policy - is not in the service at all. It lives in a separate library called aioz-common, which every service imports.

So there are three places code can be, and knowing which is which is most of finding your way around:

Where	What is in it	Example
`aioz-common`	Infrastructure every service needs. No business rules	Redis client, JWT signing, rate limiter
`internal/` in the service	This service's features and wiring. Private - nothing else can import it	Billing rules, the HTTP handlers
`pkg/` in the service	The database adapters implementing what `internal/` asked for	The gorm repositories

Two paths through the code

Almost everything a service does is one of these two, and they converge in the same place.

A request arrives.

HTTP  ->  internal/app/<feature>/endpoint.go   parse, validate, format the reply
      ->  internal/app/<feature>/service.go    the business rules
      ->  internal/app/<feature>/store.go      an interface: "I need to save this"
      ->  pkg/v1/repositories/                 the gorm code that actually does
      ->  Postgres

A scheduled job fires. It skips the top layer and joins at the second:

timer ->  internal/app/<feature>/chore.go      "every 5 minutes, do this"
      ->  internal/app/<feature>/service.go    the same business rules
      ->  ...                                  the same path down

Two things follow from this picture, and they are the rules people most often get wrong at first:

service.go never knows how it was called. No echo, no HTTP request, no knowledge that a route exists. That is what lets a chore and a handler share it, and what lets a test drive it without a server.
service.go never imports the gorm code. It depends on the interface in store.go; the composition root passes in the real implementation. That is what lets a test run without a database.

Arrows point one way the whole distance down. Nothing calls back up.

Your first hour

Clone a service and open these four things, in this order:

cmd/<service>/main.go - the list of roles. Whatever this service can do, it is one of the subcommands here.
internal/cli/api/ - what the api role actually starts.
internal/app/ - one directory per feature. Pick the smallest one and read it end to end. Every feature is laid out the same way, so having read one you can read all of them.
internal/app/<something>/service.go - the business rules, with no HTTP and no SQL in sight. This is where the service actually decides things.

Two questions worth being able to answer by the end of it: where does an HTTP request enter and what does it touch on the way through, and what is running when nobody is making a request.

The two reference repositories

Two services already have this shape, and they are what everything else is being moved towards:

aioz-stream (gitlab.internal/aioz-stream/aioz-stream) - the larger of the two, with two dozen features and its own SOURCE_STRUCTURE.md at the repository root spelling out every rule with examples from the tree. Read that one when you need the detail.
parser-service (gitlab.internal/aioz-map/parser-service) - smaller and easier to take in whole. The better first read.

This document is the shared contract between them, and what every other backend adopts.

Why one shape

A backend that is laid out like every other backend is one a person can open cold. The specific rules below buy specific things, and the reason is worth more than the rule:

Vertical slices, not horizontal layers. A feature's handlers, business logic, wire types, background work and store contracts live in one package. The alternative - controllers/, services/, repositories/, models/ - spreads one feature across five directories and makes every package a shared surface anything can reach into.
One binary, several roles. The API, the background jobs and the queue workers are the same image with different subcommands. One build, one config file, one set of migrations, and a role can be scaled or restarted on its own.
Dependencies point one way. Transport calls business logic, business logic calls a store contract, nothing calls back up. A slice never imports a sibling to reach its concrete types.

Layout

cmd/<service>/
  main.go                  One binary. Subcommands are the roles, below.

internal/
  app/                     Every vertical slice, plus the composition root.
    <slice>/                 One feature area. See "A slice".
    server/                  HTTP transport and the Module contract.
  cli/                     What each subcommand runs.
    api/ core/ consumer/     One package per role.
    migrate/ setup/          Schema and one-shot commands.
    lock.go                  The single-instance flock.
  domain/                  Entities shared by two or more slices.
  store/                   Store contracts shared by two or more slices.
  config/                  Configuration parsing and validation.
  middlewares/             Auth, rate limiting, request logging.
  migrations/              golang-migrate files. See ci-cd.md.
  utils/                   Cross-cutting helpers. No business logic.
  tools/                   Repository invariants, enforced as tests.

pkg/v1/repositories/       The gorm adapters implementing the store contracts.
deploy/                    Dockerfiles, compose, deploy.sh, haproxy. See release.md.

A helper used by one slice stays in that slice. It moves to internal/utils/ on the second consumer, not in anticipation of one. Same rule for entities and store contracts: one slice, keep it local; two or more, promote it to internal/domain/ or internal/store/.

Shared code: aioz-common

aioz-common (gitlab.internal/tuan.quang.tran/aioz-common) is one Go module holding the infrastructure every service needs. One module, one release cycle, one CI pipeline, so a service adds a single require line instead of tracking several repositories.

A rough map of what is in there, so you recognise the import when you see it:

Area	Packages
Process plumbing	`config`, `logging`, `logger`, `lifecycle`, `version`, `signal`, `proclock`
Data stores	`postgres`, `redis`, `rabbitmq`, `lrucache`
Security	`jwt`, `secret`, `password`, `httpsec`, `ratelimit`, `clientip`
Errors and control flow	`apierr`, `errs2`, `retry`, `cycle`, `context2`
Values	`uuid`, `currency`, `memory`, `date`, `period`, `time2`
Outbound	`httpclient`, `mail`, `slack`, `upload`, `depin`

Every package has its own README.md, and they are worth reading before you write the thing they already do. Several of them exist because the obvious version is subtly wrong: httpclient because http.DefaultClient has no timeout, postgres because pgxpool.New returns a pool that has not connected to anything yet, jwt because trusting the token header's alg is how algorithm-confusion attacks work.

What belongs in it

Yes: infrastructure, wire formats, and values with no opinion about our business. A Redis client. A money type. A backoff policy.

No: anything that knows what a stream, a workspace or an invoice is. The moment a package in aioz-common needs a domain concept, it belongs in the service instead.

Reaching for it

Before writing a client, a cache, a token or a retry loop, check whether aioz-common already has one. It usually does, and the version there has had the edge cases found for it already.
Needing a change in aioz-common is normal. Change it there and release a version - do not fork the behaviour into your service because the shared one is almost right.
Pin the version in go.mod like any other dependency. A service upgrades when it chooses to, not when the library moves.

Roles

One binary, one config file, one image. The subcommand picks the role:

Role	Runs	Instances
`api`	The HTTP transport and everything it serves	As many as you like, behind HAProxy
`core`	The background jobs: chores and scheduled work	Exactly one
`consumer`	Queue workers, one broker connection, N handlers	As many as you like
`migrate`	`up`, `down`, `force`, `version` - the schema. Run by CI through `deploy.sh migrate`, in its own job before deploy; never by a person and never by another role at startup	One, from the pipeline
`setup`	One-shot bootstrap: seed data, key generation	By hand

Giving each role its own process, rather than running chores inside the API, is what makes the other rules hold:

The API can scale. Three API replicas behind HAProxy do not run a scheduled job three times.
A slow job does not slow requests. They are separate processes with separate memory and separate CPU limits.
Restarting one does not restart the others. A stuck consumer is restarted without dropping a request.
Each builds only what it needs. core needs a database. It does not need token signing keys, a mail sender or a gRPC peer, and its wiring says so.

`core`

core is where the background jobs run. Anything on a schedule - a receipt run, a view-count rollup, an expiry sweep, a reconciliation pass - is a chore registered on the core app and nowhere else.

Exactly one instance. Two core processes means every chore runs twice, which for anything that writes is a bug that shows up as duplicated rows weeks later. Two things enforce it:

a flock in core.lock_dir, held for the process's life, so a second core on the same host fails to start instead of quietly doubling the work;
the deploy, which never runs two core containers at once. See deploying a role.

The flock is per host, so it is a guard rail, not a distributed lock. If a chore ever has to be safe across hosts, it takes an advisory lock in Postgres and the chore says so in its doc comment.

A slice

A slice is a package under internal/app/. Files follow one naming convention, so any slice reads without learning its layout first:

File	Holds
`common.go`	Package doc comment and the slice's `Error` tag
`endpoint.go`	The `Endpoint`: HTTP handlers, `Name()`, `Routes()`
`service.go`	The `Service`: business logic. No HTTP, no gorm
`store.go`	The store contracts this slice owns, as interfaces
`model.go`	Entities belonging to this slice alone
`dto.go`	Requests and responses, beside the handlers that serve them
`chore.go`	Background work: constructors returning a `*chore.Chore`
`consumer.go`	Queue workers

Not every slice has every file. A slice with no background work has no chore.go.

common.go declares an error tag naming the slice, so a log line says where a failure came from without a stack trace:

// Package usage is the usage-and-billing slice: what a workspace has consumed,
// what it has been charged, and the scheduled work that keeps both current.
package usage

import "github.com/zeebo/errs/v2"

// Error tags every failure that originates in this slice.
var Error = errs.Tag("usage")

A slice that grows several distinct feature areas nests them as sub-slices, each a full slice in its own right; the parent then owns composition and nothing else. Nest when each area carries its own handlers, DTOs and store contract - not to hold one type.

Dependency rules

Allowed:

cmd → cli → app (composition root) → slice
Endpoint → Service → store contract → (implemented by) pkg/v1/repositories
Chore    → Service → store contract
any layer → internal/domain, internal/utils, internal/middlewares
parent slice → its own sub-slices

Forbidden:

Service  → Endpoint              business logic must not know about protocols
Service  → Chore                 business logic must not know about scheduling
Service  → pkg/v1/repositories   depend on the contract, not the gorm adapter
Endpoint → Endpoint              slices are independent
Chore    → Chore                 chores are independent
slice    → sibling slice         declare a narrow interface instead
sub-slice → its parent           the parent imports the child, never the reverse

When a slice needs something a sibling owns, it declares the narrow interface it needs, in its own package, next to the consumer:

// StreamKeys is the part of the stream-key service this package needs: a
// recording always hangs off a key, and the key's owner is what authorizes
// the call.
type StreamKeys interface {
    GetLiveStreamKeyById(
        ctx context.Context, id uuid.UUID, authInfo domain.AuthenticationInfo,
    ) (*domain.LiveStreamKey, error)
}

The sibling's service satisfies it structurally, the composition root passes it in, neither package imports the other, and the interface documents exactly how much of the sibling this code relies on - usually two methods out of thirty.

Lifecycle and drain

Every long-running component registers on the app's lifecycle group by name. Registration by name rather than a bare go buys three things, each of which only matters when something is wrong: a failure logs which component stopped, every goroutine carries a pprof label so a dump names the chore instead of an anonymous closure, and a component that hangs on the way down is named in the timeout message with its stack.

Order matters. Chores and consumers register before the API so they close after it: the transport stops taking new requests first, and nothing is still producing work when the thing it depends on goes away.

The drain state

On SIGTERM a process does not go straight to cancelled. It has three states:

serving  --SIGTERM-->  draining  --drain_timeout-->  stopping

State	Accepts new work	Work in flight
`serving`	Yes	Runs
`draining`	No	Runs, with an uncancelled context
`stopping`	No	Context cancelled, requeued, process exits

The rule that matters:

Entering draining stops the process taking new work. It does not cancel the work already running. Only the hard deadline does that.

Why

The obvious implementation - one context, cancelled on SIGTERM, handed to everything - throws away the attempt in flight. A chore that is eight minutes into a ten-minute reconciliation pass is killed at eight. A consumer handler that has done the expensive half of its work and not yet committed is killed and the message requeued, so the next instance starts from zero. Worse, if the retry policy counts that as an attempt, a deploy quietly spends one of the three a message gets, and a message unlucky enough to be in flight across three deploys is dead-lettered having never actually failed.

None of that is visible in a graph. It shows up as a job that "sometimes takes two runs", which is the kind of bug nobody files.

Draining costs a few seconds of shutdown and removes the whole class.

What each role does when it drains

Role	On entering `draining`
`api`	Stop accepting new connections. `/health` returns 503 so HAProxy drains it. In-flight requests finish
`core`	Every chore loop stops scheduling the next pass. The pass currently running keeps its original context and runs to completion
`consumer`	Send `basic.cancel` on the AMQP channel, so the broker delivers nothing more. Nack anything already buffered and not yet started, straight away, so another instance gets it immediately. Handlers already running keep their original context

Then the process waits, up to shutdown.drain_timeout, for the in-flight work to finish. What is still running when that expires gets its context cancelled, is requeued rather than acked, and the process exits. A drain that hits the timeout logs the component and the count - it means the timeout is too short or a job is too long to be one unit.

The contract

A component that can hold work implements:

type Runner interface {
    // Name identifies the component in logs, in a goroutine dump, and in the
    // shutdown sequence.
    Name() string

    // Run does the work. Its context is canceled only at the hard deadline,
    // never on SIGTERM.
    Run(ctx context.Context) error

    // Drain stops the component accepting new work and returns when the work
    // in flight has finished or ctx - the drain deadline - expires.
    Drain(ctx context.Context) error

    // Close releases resources. Called after Drain returns.
    Close() error
}

Two contexts, and keeping them apart is the whole mechanism:

the work context, which lives as long as the process and is cancelled only at the hard deadline;
the drain signal, delivered by Drain, which says stop accepting.

Handing the work context to a handler and the drain signal to the loop that feeds it is the difference between finishing the last job and wasting it.

Config, under shutdown:

Key	Default	Meaning
`drain_timeout`	`30s`	How long `draining` waits for in-flight work
`shutdown_timeout`	`drain_timeout + 10s`	Hard deadline. The process exits regardless

drain_timeout must be longer than the slowest normal unit of work and shorter than the deploy's own timeout, or the deploy kills the container mid-drain and nothing was gained. A chore whose single pass is longer than drain_timeout is too coarse: make its pass batched and resumable rather than raising the timeout past the deploy's.

Deploying a role

This is where drain meets release.md.

api is blue/green. The new slot starts and passes its health check before HAProxy moves traffic; the old slot drains behind HAProxy's own drain. No gap.
core cannot be blue/green. There must be exactly one, and the flock will refuse a second, so the deploy is: SIGTERM the old container, wait for it to drain and exit, then start the new one. The gap is the drain, and during it no scheduled job runs - which is fine, because a chore that cannot miss one interval is a chore that needs a durable queue, not a shorter gap.
consumer can overlap: start the new one, then SIGTERM the old. The broker holds anything neither is ready for. Handlers must be idempotent anyway, because a requeue after a hard deadline can deliver a message twice.

Adopting

For a backend not yet on this shape, in this order. Each step is a separate merge request, and the service is releasable after every one.

cmd/<service>/main.go with subcommands. Even if core and consumer start empty. This is the step everything else hangs off.
Move migrations to MIGRATIONS_DIR and out of startup. m.Up() at boot races across replicas and turns a bad migration into an outage instead of a failed pipeline. See ci-cd.md.
Pull background work out of the API into core. Add the flock.
Implement drain. The Runner contract above, the two contexts, and the shutdown config block. Do this before splitting slices - it is small, self-contained, and it is the part that silently costs work today.
Split the first slice. Pick the smallest feature area, give it common.go with its error tag, then store.go, service.go, dto.go, endpoint.go. Do not convert the whole repository at once.
Move the rest a slice at a time, with the old layout still compiling beside the new one.

Order of repositories: parser-service and aioz-stream are already there. Then backend (aioz-map), then hub, then crawler-service.

Source structure ​

The shortest version ​

Two paths through the code ​

Your first hour ​

The two reference repositories ​

Why one shape ​

Layout ​

Shared code: aioz-common ​

What belongs in it ​

Reaching for it ​

Roles ​

core ​

A slice ​

Dependency rules ​

Lifecycle and drain ​

The drain state ​

Why ​

What each role does when it drains ​

The contract ​

Deploying a role ​

Adopting ​