ops: deploy the server from GHCR, and a verified restore drill

[kipavy]

Preparation for growth on the single-box setup. Nothing here is applied to production — the compose change takes effect only when the container is next recreated, and the drill it documents has already been run.

Deploy from GHCR instead of building on the host

compose.prod.yml built the image on the production host from a git checkout living inside a Docker volume. That tree drifted from every other copy of the repo five separate times: dirty trees at ancient HEADs, a commit that existed in no other repo, and a stale feature branch whose merge would have reverted a production fix. Each build takes 13–17 minutes competing with the live server for this host's two cores, and that checkout's .git/config holds a plaintext GitHub PAT.

.github/workflows/docker.yml already publishes a multi-arch image on every push to main, so production now pulls it:

image: ${SERVER_IMAGE:-ghcr.io/voltiusapp/voltius-server}:${SERVER_TAG:?pin an immutable sha-<short> tag, never latest}

SERVER_TAG has no default on purpose. latest is mutable, overlapping pushes race on it, and it cannot be rolled back to. With no tag set, up -d refuses to start rather than deploying something unidentified.

Also adds restart: unless-stopped, which this service was missing. It was the only container in the stack with a restart policy of no, so a host reboot left the API down until someone noticed.

Verified before opening this: ghcr.io/voltiusapp/voltius-server:sha-469337c exists with both linux/arm64 and linux/amd64, and the deployment tree's tracked files are clean at 469337c, which is origin/main. The cutover is therefore content-neutral — the same code, from a different source.

Runbooks

docs/runbooks/deploy-server.md covers the pinned-tag deploy, rollback, and the first cutover from the local-build setup. It records the traps that have each cost time before: --env-file is mandatory or the server starts with blank secrets and fails at the first authenticated request; an unauthenticated request to a route returns 401 if it exists and 404 if the image lacks it, which proves route registration without writing production data; and a rollback reverts the binary but never a migration.

docs/runbooks/restore-database.md documents a restore drill actually run against the R2 archive on 2026-09-21 — the first time that archive had ever been restored:

Measure	Result
Base fetch	6 s, 77 MB
WAL replay to end of archive	68 s
Total to accepting connections	~95 s
Recovery point	last transaction ~5 s before the drill began
Schema	32 tables, migration 42, 0 failed
Cross-check	table set identical to the independent nightly pg_dump

It was run twice, the second time by executing the runbook's own commands verbatim, so the document is tested rather than written from memory.

Two ways a drill can damage production are called out explicitly, because both are silent: a restored copy promoted with archive_mode=on pushes its timeline-2 WAL into the shared R2 prefix that production depends on, and a drill container attached to the cloudflare network answers to the name voltius-db.

One incidental finding: the restored database is 54 MB, of which sync_blobs is 44 MB. That accounts for the nightly dump growing 15 MB to 32 MB over five weeks — payload, not an unpruned metadata table.

Applying it

1. SERVER_TAG=sha-<short> into .env.dockhand.
2. Tag the last locally built image as a fallback.
3. Copy this compose.prod.yml into the deployment tree.
4. pull, then up -d server, then check health and the 401.