ops: bring the database stack into the repo, and make its image portable

[kipavy]

The Postgres half of production existed only as files in one directory on one host. A grep for wal-g across this repo matched exactly one file — the restore runbook added last week. So the runbook could restore a backup while nothing in git could recreate the thing that takes them.

What was missing, concretely:

• The durability configuration. wal_level=replica, archive_mode=on, archive_command=wal-g wal-push %p, archive_timeout=60, max_wal_senders=3. The recovery point objective is that fourth flag and nothing else, and it was one untracked line.
• wal-g delete retain FULL 14 in the base-backup loop. That is the fix for the incident where the bucket reached 8.29 GiB backing a 54 MB database. Rebuild the stack from memory without it and the regression returns silently, visible only as a bill.
• The image wal-g runs in, and the reasoning behind the asset it downloads.
• The dump sidecar's retention, the rclone mirror, and the R2_* → AWS_* mapping WAL-G needs.

Portability

The old Dockerfile hardcoded wal-g-pg-20.04-aarch64 with no checksum. On an amd64 host that image builds successfully and then cannot execute wal-g — the failure arrives at the first archive push, not at build time.

It now picks the asset from TARGETARCH, verifies the .sha256 upstream publishes, and does the download in an alpine stage, so the final image carries no package manager, no curl, and no apt lists. Verified both legs: the amd64 fetch produces an EM_X86_64 ELF, and a native build runs wal-g v3.0.8.

Base pinned to a suite

postgres:17 floats between Debian releases. glibc owns text collation, initdb bakes it into every text index, and moving a data directory across glibc versions can reorder them. Pinned to trixie, which is what the running cluster is on (glibc 2.41, checked against the live container).

For the record: an alpine base builds and passes a full restore drill with gcompat, and would save about 250 MB. Not taken — musl collation differs from glibc, so it cannot be swapped under an existing data directory, only across a dump and reload.

Studio

Folded in behind a compose profile with restart: "no", so up -d never starts it and a reboot cannot bring it back:

docker compose -f compose.db.yml --env-file .env.db --profile studio up -d

Confirmed that config --services lists four services by default and six with the profile. Its demo keys move to the env file instead of being committed.

Also

The dev compose.yml ran postgres:16-alpine against a production on 17.

Nothing here touches the running stack: it keeps its own copy of these files and its own image. Adopting the repo copy is a separate, optional step, and does not require rebuilding the image.

[kipavy]

Checked against the README and the docs site before merging, and one thing in here did break them — fixed in 3c33d90.

The postgres:16-alpine → 17-alpine bump was a real break. The self-hosting quickstart tells people to update with docker compose pull && docker compose up -d. On an existing volume that recreates the database against a data directory initialised by 16, and Postgres 17 refuses:

FATAL:  database files are incompatible with server

Reproduced it directly (init a volume with 16-alpine, start 17-alpine on it). Reverted to 16, now behind POSTGRES_IMAGE_TAG for anyone who wants to move across a dump and reload. Nothing in the docs pins a Postgres version, so the compose file was the only place this leaked.

Then ran the documented quickstart end to end in isolation, with the current latest image: /health → ok, /v1/meta → {"self_hosted":true,"billing_enabled":false}, 42 migrations applied, Postgres 16.14. Matches what the quickstart page promises.

Alpine, as far as it goes safely

PG_BASE now selects the base and gcompat is installed when it is alpine, so one Dockerfile covers both. 496 MB against 742 MB.

Tested the whole archive path on musl rather than just a restore: a fresh alpine cluster with archive_mode=on and archive_command=wal-g wal-push, a backup-push, 1000 more rows and a WAL switch after that backup, then backup-fetch + replay + promote into a second container — which came back with all 6000 rows, including the 1000 written after the base backup. Into a throwaway prefix in the bucket, purged afterwards; only voltius-prod/ remains.

The default stays glibc because production is glibc, and the two sort text differently:

base	libc	size	same ORDER BY
postgres:17-trixie	glibc	742 MB	10,2,a,A,b,B,eclair,éclair,_x,Zoe
postgres:17-alpine	musl	496 MB	10,2,A,B,Zoe,_x,a,b,eclair,éclair

initdb bakes that ordering into every text index, so it is a dump-and-reload decision, not an image swap. Both rows are in .env.db.example next to the setting.

Everything else in the stack is already alpine: the server runtime and its builder, the dump sidecar, rclone 1.69 (Alpine 3.21), and the wal-g fetch stage.