Harden prod Docker image: drop toolchain, read-only rootfs, least privilege#37
Open
radiantnode wants to merge 10 commits into
Open
Harden prod Docker image: drop toolchain, read-only rootfs, least privilege#37radiantnode wants to merge 10 commits into
radiantnode wants to merge 10 commits into
Conversation
…vilege Shrink the runtime attack surface and run the prod stack with the smallest possible privileges and a read-only root filesystem. Dockerfile: - Split the Python image into a builder stage (build-essential + cmake, which compile asyncpg/blspy into a self-contained venv) and a clean runtime stage that copies only that venv. The compiler, headers, and apt metadata no longer ship in the runtime image — the single biggest piece of runtime surface. - Copy only the runtime files (main.py, server/, migrations/, static/) instead of the whole repo; scripts/, ops/, tools/, tests/, loadtest.py, and the build/compose files stay out of the image. - Strip every setuid/setgid bit so a compromised process can't reuse a privileged helper to escalate. - Switch the nginx stage to nginxinc/nginx-unprivileged: the master runs as a non-root user (uid 101) on a high port, so the container needs no caps and can run read-only. ops/nginx.conf: listen on 8080 (non-root), route the pid, all temp dirs, and logs to /tmp + stdout/stderr so the root FS can be read-only. docker-compose.prod.yml: - web + nginx: read_only root FS with a small /tmp tmpfs, cap_drop ALL, no-new-privileges; web also gets init: true and an HTTP healthcheck. nginx published port now maps to 8080. - redis (persistence already off), prometheus, postgres_exporter: read_only, cap_drop ALL, no-new-privileges; redis runs as the redis user to avoid needing CAP_SET[UG]ID. - postgres + grafana: no-new-privileges (their privilege-dropping entrypoints and writable state make a full read-only/cap-drop pass a separate change). Verified: both images build with no compiler present and no setuid binaries; web (327MB) and nginx serve 200 on a read-only rootfs with all caps dropped and no-new-privileges; nginx runs as uid 101 and serves /static; prod compose parses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mjorQqBtf4Jm5VUvVRLhX
Builds on the image hardening with runtime least-privilege controls.
Resource + pid limits (DoS containment): every service now caps cpus, memory,
and pids_limit (top-level keys, honored by `docker compose up`), so a runaway or
compromised container can't exhaust host CPU/RAM or fork-bomb the box.
Secrets as files, not env vars: app-owned secrets (JWT_SECRET, STATS_TOKEN,
METRICS_TOKEN) are delivered to the web container as files under /run/secrets via
compose `secrets:`, so they no longer appear in `docker inspect` or
/proc/<pid>/environ. server/config.py gains a `_secret()` helper that reads
`{NAME}_FILE` (a path) first, falling back to the `{NAME}` env var then a default
— an unreadable/empty file falls back rather than silently blanking a secret.
The helper is also applied (opt-in) to REDIS_URL, POSTGRES_DSN, GRAFANA_PASS, and
DISCORD_BOT_TOKEN so operators can file-back those too; the redis/postgres/grafana
passwords stay env-driven for now since those images/URLs consume them directly.
tmpfs hardening: the read-only containers' one writable path (/tmp) is mounted
noexec,nosuid,nodev so it can't be used to stage and execute a dropped binary.
.env.prod.example documents the new secret-file flow (materialise the three files
under the gitignored ops/secrets/ before `up`).
Verified: prod compose renders valid; the web image serves 200 on a read-only
rootfs with a noexec /tmp, pids cap, and JWT loaded from a mounted file (absent
from the container env); _secret() file/env/default fallbacks confirmed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mjorQqBtf4Jm5VUvVRLhX
… deps, image-signing CI
Completes the deployment hardening pass.
Network egress split: a new two-network topology. `internal` is egress-disabled
(internal: true) and holds the data stores — redis, postgres, postgres_exporter,
prometheus — so a compromised one has no route to the internet (can't exfiltrate
or pull a second stage). `edge` (the only network with a gateway) is joined only
by web (drand/Discord egress), nginx (public ingress port), and grafana (admin UI
port; its phone-home is disabled via GF_ANALYTICS_*). web sits on both.
Full read-only + cap-drop for the last two services:
- postgres: read_only root with the data dir on its volume and /tmp + the socket
dir on tmpfs; cap_drop ALL + the minimal cap_add the root entrypoint needs to
chown the data dir and gosu-drop (CHOWN, DAC_OVERRIDE, FOWNER, SETGID, SETUID).
- grafana: read_only root with /tmp tmpfs (writes its sqlite/plugins to the
volume); cap_drop ALL (already non-root).
Every service is now read_only with all caps dropped (postgres keeps the minimal
add-back), no-new-privileges, and cpu/memory/pids caps.
All credentials are now mounted files, never env vars (so none appear in
`docker inspect` / /proc/<pid>/environ):
- redis reads its password from a secret and writes a tiny config on tmpfs, so
the password never lands in argv (vs --requirepass on the command line).
- postgres uses POSTGRES_PASSWORD_FILE; postgres_exporter uses the split
DATA_SOURCE_URI/USER + DATA_SOURCE_PASS_FILE form.
- grafana uses GF_SECURITY_ADMIN_PASSWORD__FILE and reads the datasource
password via the $__file{} provisioning provider.
- web reads the full redis URL / postgres DSN / grafana pass from files
(REDIS_URL_FILE etc.) via config._secret(), alongside jwt/stats/metrics.
Hashed, reproducible dependency install: requirements.lock is regenerated from
requirements.txt with pip-compile --generate-hashes (it was stale and unhashed —
e.g. asyncpg 0.30 vs the declared 0.31). The Dockerfile installs it with
--require-hashes so a tampered or MITM'd index can't substitute an artifact.
Supply-chain CI (.github/workflows/image-security.yml): every PR builds the web
image, fails on fixable HIGH/CRITICAL CVEs (Trivy), and uploads an SPDX SBOM
(Syft). On main, both images are pushed to GHCR with SLSA provenance + SBOM
attestations and keyless-signed with cosign (Sigstore OIDC, no long-lived keys).
Verified end-to-end: brought up the full prod compose with generated secret
files — postgres + redis reach Healthy (read-only postgres, file passwords work),
migrations apply, web serves 200 through nginx, grafana serves 200 (datasource
provisioned via $__file{}). Confirmed: the four data stores are on `internal`
only with NO default route (egress blocked) while edge has one; web's env exposes
only *_FILE pointers (no credential values); the redis password is absent from
its process argv; all seven containers are ReadonlyRootfs with 0 restarts. The
web image builds with --require-hashes (all 37 packages hash-verified); prod
compose renders valid; ruff passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mjorQqBtf4Jm5VUvVRLhX
aquasecurity/trivy-action tags are v-prefixed (v0.29.0 … v0.36.0); the unprefixed 0.29.0 doesn't resolve, so the scan job failed at action setup. Pin to v0.36.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mjorQqBtf4Jm5VUvVRLhX
The new image-security scan (Trivy, fail on fixable HIGH/CRITICAL) flagged the base image's OS packages: a CRITICAL openssl heap overflow (CVE-2026-31789), several HIGH openssl RCE/UAF/DoS issues, and perl-base — all with fixed Debian versions available. The pinned python:3.12.8-slim-bookworm tag lags Debian's security repo, so a fresh build still ships these. Add `apt-get update && apt-get -y upgrade` (apt lists removed after) to the web runtime stage. Verified locally: openssl/libssl3 -> 3.0.20-1~deb12u2 and perl-base -> 5.36.0-7+deb12u3 (exactly the fixed versions), and a local Trivy run (HIGH,CRITICAL, --ignore-unfixed) now reports 0 findings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mjorQqBtf4Jm5VUvVRLhX
CodeQL "Clear-text logging of sensitive information" (high) flagged gamestore.py: the connect log emitted REDIS_URL verbatim, which embeds the password (redis://:PASSWORD@host). Surfaced once config._secret() made CodeQL treat REDIS_URL as sensitive, but it was a real leak — the password landed in the app logs on every startup. Drop the URL from the success log (log a plain "connected" line; the host is static config and the confirmation is the useful signal). The failure path keeps an actionable target via a new _redact() helper that strips credentials (scheme://host:port only) — used in the RuntimeError message, which is not a logging sink. No REDIS_URL-derived value reaches a log sink anymore. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mjorQqBtf4Jm5VUvVRLhX
Core end-to-end suite (gameplay, multiplayer sync, pause, player + host reconnect, WebAuthn auth) run against docker-compose.prod.yml — read-only containers, file-based secrets, egress-split networks, nginx-unprivileged. 24/24 executed checks passed; hardening properties confirmed live (no Redis password in logs, read-only Postgres accepted the WebAuthn user write, data stores have no egress, no secret values in the web env). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mjorQqBtf4Jm5VUvVRLhX
Full 18-step telemetry pipeline test run against docker-compose.prod.yml —
token-gated /metrics, internal-only Prometheus, Grafana datasource password via
$__file{} on a read-only container, file-based secrets. 18/18 passed: Prometheus
scrapes the gated /metrics via the mounted token file over the egress-split
network (rolls 99=99 exact), Postgres datasource connects, all 5 dashboards
render error-free, 0 telemetry drops / 0 live-push failures.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mjorQqBtf4Jm5VUvVRLhX
…ed-stack log Three additional checks on the hardened prod stack, all PASS: - Host transfer after 60s DISCONNECT_GRACE: role reassigns to the remaining player. - Read-only Postgres restart on an existing volume: comes up via the non-initdb path under ReadonlyRootfs with data intact and password-from-file. - Scale to 3 read-only/capped replicas: cross-instance sharing works (create on web-2, join on web-1, roll fans out across them via Redis pub/sub over the egress-split internal net); 0 restarts / no OOM. A single-IP synthetic flood was correctly throttled by the per-IP abuse caps (not a throughput benchmark). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mjorQqBtf4Jm5VUvVRLhX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Shrinks the prod runtime attack surface and runs the stack with the smallest possible privileges and a read-only root filesystem.
Dockerfile
build-essential+cmake(which compileasyncpg/blspy) now live only in a throwaway builder stage that produces a self-contained venv; the runtime stage copies just that venv. The compiler, headers, and apt metadata no longer ship — the single largest piece of runtime surface.main.py,server/,migrations/,static/ship.scripts/,ops/,tools/,tests/,loadtest.py, and the build/compose files stay out of the image.nginxinc/nginx-unprivileged: the master runs as a non-root user (uid 101) on a high port, so the container needs no caps and can run read-only (this was the last root process in the stack).ops/nginx.conf
8080(non-root) and route the pid, all temp dirs, and logs to/tmp+ stdout/stderr so the root FS can be read-only.docker-compose.prod.yml
read_onlyroot FS with a small/tmptmpfs,cap_drop: [ALL],no-new-privileges;webalso getsinit: trueand an HTTP healthcheck. The nginx published port now maps to8080.read_only,cap_drop: [ALL],no-new-privileges; redis runs as theredisuser to avoid needingCAP_SET[UG]ID.no-new-privilegesonly — their privilege-dropping entrypoints and writable state make a full read-only/cap-drop pass a separate, test-heavy change (noted inline).Verification
web(327 MB) andnginxagainst real Redis/Postgres with--read-only --cap-drop ALL --security-opt no-new-privileges:webserves/→ 200,nginxproxies/→ 200 and serves/static→ 200, running as uid 101.docker compose -f docker-compose.prod.yml configparses.Follow-ups (intentionally out of scope)
cap_addsubset + tmpfs tuning — deserves its own tested change).🤖 Generated with Claude Code
https://claude.ai/code/session_015mjorQqBtf4Jm5VUvVRLhX
Generated by Claude Code