Repository navigation
Phase E — Production wiring + staging validation + rollout/rollback runbook #100
Description
Activity
Phase E — first-session plan (2026-05-20)
Kicking off Phase E. This is the long-pole that unblocks the unified core going public → K9 Dogfood green → idaptik#77 closable. Multi-session work over weeks; no code PR opens until this plan is on the issue.
Live channel state (re-checked, not recalled)
- Phase A (Phase A — Contract definition + policy-authoring workflow + example Verb Governance Spec #96) — merged via boj-server#78 (contract + workflow + example policy)
- Phase B (Phase B — mTLS as primary trust-level path (Cowboy TLS + real-CA fixture) #97) — merged via http-capability-gateway#10 (Cowboy TLS + real-CA fixture, mTLS-as-primary)
- Phase C (Phase C — E2E verification + gateway↔gnosis seam tests #98) — merged via http-capability-gateway#11 + boj-server#90 + boj-server#93 (E2E + seam tests + §3 invariant 3 documented)
- Phase D (Phase D — Benchmark formalisation + published latency + CI regression alert #99) — scaffold merged via http-capability-gateway#12 (2026-05-20T08:24Z). Real numbers explicitly deferred to D-2..D-4 (separate PRs);
bench/baseline.json _status: "scaffold-placeholder"keeps the perf-regression gate non-blocking until populated. - Phase E (Phase E — Production wiring + staging validation + rollout/rollback runbook #100) — this issue. Starting now.
Phase E acceptance (restated for the record)
Trustfile.a2ml [CLOUDFLARE_EDGE_SECURITY].rate_limiting.tier_2_gateway.statusflipped from"PENDING — http-capability-gateway wiring forthcoming"(currently line 900 of.machine_readable/contractiles/trust/Trustfile.a2ml) to a real production value.- HCG-in-front-of-BoJ wired in staging; staging health green under load that matches Phase D benchmark numbers.
- Rollout-and-rollback runbook lives in the BoJ repo at
docs/runbooks/hcg-tier2-rollout.adoc(placement TBD — plan §E5 saysdocs/integration/gateway-rollback-runbook.md; I'll align on whichever exists / propose if neither). - K9 Dogfood gate flips red → green in BoJ.
- HCG is the externally addressable surface; BoJ no longer reachable directly.
Single-channel discipline — what Phase E can and cannot do now
Phase D gates Phase E production rollout (plan §D "Blocks: Phase E production deployment requires D3 (regression alert) to be in place before rollout"). It does not gate Phase E scoping, runbook drafting, or audit of consumer-side assumptions. So this session will:
- ✅ Drive — design/runbook/audit work that is risk-free w.r.t. Phase D state.
- ❌ Hold — staging traffic-shift, production flip, and Trustfile.a2ml
status: "DEPLOYED"flip until Phase D-3 (regression alert armed) and D-4 (real baseline numbers) land. Out-of-phase work is prohibited.
First work-item: rollout/rollback runbook draft (E5-shaped)
I'm proposing E5 (runbook) lands first because:
- It captures the design while context is fresh and is the lowest-risk artefact in Phase E.
- It's a pure-documentation PR — no code, no CI risk, no Phase D dependency.
- It sets the operational vocabulary (traffic-shift mechanics, rollback triggers, dashboard names) that the later E2/E3/E4 PRs will reference.
- Several details (on-call rota, prod cert material, dashboard URLs) need owner input — drafting now surfaces those gaps early so the owner can fill them async rather than blocking a code PR mid-rollout.
PR will be
Refs standards#100andRefs standards#91(neverCloses— joint-close is owner-only).Second work-item (parallel-safe): consumer-side audit
Before E1/E2 wiring PRs, I need a written catalogue of every BoJ site that currently assumes internal/loopback addressing. This is read-only and can be filed as a comment / draft note on #100 alongside the runbook PR. Outputs will include:
- Cowboy bind addresses (current vs HCG-fronted target).
X-Trust-Levelingress paths in Elixir + Zig gnosis handler.- The Phase C finding:
BojRest.TrustPolicy.satisfies?/3does NOT enforce §3 invariant 3 (third clause matches regardless ofis_local). Phase C documented this; Phase E must resolve it before BoJ is reachable from non-loopback. Owner-gated one-liner from the Phase C close-out:def satisfies?(_required, _trust, false), do: false
Deferred until Phase D-3 + D-4 land
- E1 — Containerfile + k9-svc deployment spec finalisation (need confirmed runtime envelope from D-4).
- E2 — staging deployment & traffic shift (needs D-3 gate armed, D-4 numbers to compare against).
- E3 — telemetry verification under load (needs D-4 numbers to know what "good" looks like).
- E4 — production flip (gated on E2 + E3 sign-off).
What I will NOT do in this campaign
- Touch the
_wt-vclut-phase1worktree or vcl-ut PR#21 (parallel session). - Touch standards#150–[#92 Class C] mongodb-mcp: missing BsonFieldType enum in Zig FFI #155 (Class C, parallel session).
- Use
--adminto bypass red checks; CI is the merge oracle. - Close idaptik#77 — owner-gated. Will surface "now closable" in a comment when K9 turns green.
- Re-design HCG; Phase E is wiring an already-built thing.
- Add scope; drift unrelated to Phase E gets a separate issue.
Status comment cadence
Mid-week status comment on this issue summarising what landed / what's blocked / what's next. Future-me and parallel sessions need breadcrumbs.
Next action: draft the runbook PR.
🤖 Generated with Claude Code
Phase E first PR opened — boj-server#128 (E5 runbook draft)
Filed as
boj-server#128: docs(integration): HCG tier-2 rollout & rollback runbook (Phase E E5 draft) — DRAFT.Pure documentation. 308 lines covering prerequisites checklist, staging cut-over, 10/50/100% production rollout, observability, rollback (triggers + immediate-bypass + permanent-disable + post-decommission), and the final Trustfile flip.
Refs standards#100 / #91, no Closes.Single-channel discipline preserved. §1.1 of the runbook explicitly gates execution on Phase D-3 (regression alert armed) and Phase D-4 (real baseline numbers populated). Until those land, this runbook is reviewed/merged-as-design only, not executed.
Owner-input markers flag every field that needs operational context only the owner has — on-call rota, dashboard URLs, prod cert paths, traffic-shift mechanism choice, Cloudflare zone, freeze windows. Filing them now means the eventual rollout PR is not blocked mid-flight collecting answers.
Phase C §3 invariant 3 status correction. Earlier comment on
standards#91said the BoJ-side fix inBojRest.TrustPolicy.satisfies?/3was owner-gated. That comment was written beforeboj-server#106(commit40e46f6f) landed the deny clause — verified bygit logagainst the file. Phase C is fully closed; Phase E inherits a clean §3 invariant.Next planned work-item: a written catalogue of every BoJ site that currently assumes internal/loopback addressing (input to E1/E2 design). Read-only audit, will be posted as a comment here rather than as a PR.
🤖 Generated with Claude Code
BoJ-side audit — loopback/internal addressing assumptions (input to E1/E2)
Read-only audit of every site in
boj-serverthat currently assumes internal/loopback addressing for BoJ. Filed as a comment rather than a PR per the first-session plan (this is information, not a change).Findings
Already-compliant (no Phase E change needed)
elixir/lib/boj_rest/router.exlines 213–216 —loopback?/1defines IPv4 (127.x.x.x) and IPv6 (::1) loopback. Used in/cartridge/:name/invoke(line 84) and/cartridge/:name/sse(line 136). Correctly threadsis_localintocheck_trustandBojRest.CredentialDecryptor.extract/2.elixir/lib/boj_rest/trust_policy.exline 73 —def satisfies?(_required, _trust, false), do: falsealready in force (boj-server#106, commit40e46f6f). Non-loopback caller'sX-Trust-Levelis ignored.elixir/lib/boj_rest/credential_decryptor.exlines 25–60 — plaintext credentials accepted only whenis_local: true; non-loopback callers presenting plaintext credentials are rejected with an explicit "use encrypted form" reason. Sameis_localflag flow as the router.Trustfile.a2ml [SEAMS](Phase C, boj-server#90) — declares the gateway↔BoJ-gnosis seam with the loopback-only back-side bind invariant.elixir/test/credential_decryptor_test.exs+elixir/test/catalog_properties_test.exs+elixir/test/phase_c_seam_test.exs— 9-test seam coverage plus property tests on the loopback boundary. No gap on the test side.
Pre-Phase-E exposed-bind risk surfaces (need confirmation before any §3.4 decommission)
elixir/config/config.exsline 6 +elixir/lib/boj_rest/application.exline 15 — Cowboy listens onport: 7700with no explicit bind address. Plug.Cowboy defaults to:any(0.0.0.0), so without a Kubernetes NetworkPolicy / pod-network isolation the port is potentially externally addressable. Phase E §1.4 prereq is "BoJ staging instance addressable on loopback only" — this must be enforced at the container/k8s layer because the BEAM listener itself does not bind to127.0.0.1today. Action for E1/E2: either (a) add explicitip: {127,0,0,1}toPlug.Cowboy.start_linkoptions, or (b) document the NetworkPolicy requirement ink8s/deployment.yaml. ADR-0004 §1 (back-side bind isolation invariant) → contract §1 staging row says BoJ's:7700is "not externally routable" — that is currently an operational assertion, not a code-enforced one.stapeln.tomlline 100 —APP_HOST = "[::]"(all interfaces, IPv6). Same exposed-bind concern as (6). Used by the stapeln packager.k8s/deployment.yamlline 21 +k8s/service.yamllines 10–22 —containerPort: 7700exposed via Service on ports 7700 / 7701 / 7702 / 7703. Currently a ClusterIP/LoadBalancer (need to verify which) — if LoadBalancer the BoJ port is externally addressable today, defeating the §3.4 invariant before Phase E even starts. Action for E1/E2: convert the Service to a headless / ClusterIP-only address that the gateway pod can reach but external traffic cannot.gemini-extension.json,.mcp.json,smithery.yaml,README.md,k8s/deployment.yamlenv — all defaultBOJ_URL=http://localhost:7700. These are MCP-bridge / CLI consumer configs, not production ingress, so they remain correct post-rollout (the gateway lives at a different address, MCP-bridge continues to talk to BoJ loopback directly via its local sidecar). No change needed but worth flagging the dual addressing in the Phase E telemetry verification (§E3): the gateway will not see MCP-bridge traffic, so dashboards should distinguish gateway-routed vs MCP-bridge-direct traffic.
Adjacent — ADR-0013 streamable-HTTP transport (not Phase E scope but related)
docs/decisions/0013-streamable-http-transport.md— definesBOJ_HTTP_AUTHmodes (none/bearer/mtls/oidc).BOJ_HTTP_AUTH=noneis documented as "loopback only; refuses non-loopback connections". This is a parallel (non-gateway) HTTP transport that already has a loopback-only mode. No Phase E change needed — but the documentation in §1.4 of the runbook should be cross-referenced so an operator doesn't accidentally enableBOJ_HTTP_AUTH=beareron the:7700listener and treat it as Phase-E-compliant external access (it isn't; the gateway is the only Phase-E-sanctioned external surface).
Action items distilled (input to E1/E2 PRs, not to be done now)
# Site Action Phase 6 elixir/lib/boj_rest/application.ex:15Add explicit ip: {127,0,0,1}toPlug.Cowboyoptions, OR document NetworkPolicy requirementE1 7 stapeln.toml:100Change APP_HOSTto127.0.0.1for the Phase-E target deployment, OR document the container-isolation requirementE1 8 k8s/deployment.yaml+k8s/service.yamlConvert BoJ Service to ClusterIP-only / pod-internal; ensure only the gateway pod can reach :7700E1 + E2 9 MCP-bridge configs No code change. Document the dual addressing in §E3 telemetry verification E3 (in runbook §2.2 already noted) Single-channel discipline reminder
This audit is information only. The action items above are deferred until Phase D-3 + D-4 land (per the runbook §1.1 gate). When the gateway-side wiring PR is filed (E1/E2), it should reference this comment and pick up actions (6)/(7)/(8).
🤖 Generated with Claude Code
- added a commit that references this issue
on May 20, 2026 Second Phase E PR opened — boj-server#130 (audit action #6, bind-tighten)
Filed as
boj-server#130: fix(boj): bind Cowboy to 127.0.0.1 by default (HCG tier-2 E1 prereq) — ready, not draft.Materialises audit action item #6 —
Plug.Cowboypreviously bound to0.0.0.0(Plug default) with no explicitip:option, so the contract document's "BoJ's:7700is not externally routable" claim and the Phase E runbook §1.4 prereq #6 were operational assertions, not code-enforced. This PR makes the default127.0.0.1withBOJ_BIND_IPoverride (failing fast on invalid input). 7 new unit tests cover the IP parser.All 18 CI checks green. Two pre-existing
whisper-mcpcartridge property-test failures onmix test(catalog_properties_test.exs) are baseline rot onorigin/main— verified by running tests againstorigin/mainHEAD92f3b3a1in a clean detached worktree; same 2 failures, my +7 pass.Refs standards#100 / #91, no Closes. Independent of Phase D — the change is defensive hardening and ships now.Audit follow-ups still open (mentioned in PR body, deferred):
- Action chore(deps): bump the actions group with 2 updates #7:
stapeln.toml:100APP_HOST = "[::]"→ loopback. - Action chore(deps): bump the actions group across 1 directory with 4 updates #8:
k8s/deployment.yaml+k8s/service.yamlService → ClusterIP-only.
Will pick these up next unless the owner wants a different next step.
🤖 Generated with Claude Code
- Action chore(deps): bump the actions group with 2 updates #7:
Third Phase E PR opened — boj-server#131 (audit action #8, k8s ClusterIP)
Filed as
boj-server#131: fix(k8s): Service for BoJ to ClusterIP (HCG tier-2 E1 prereq) — DRAFT.Materialises audit action item #8 —
k8s/service.yamldeclaredtype: LoadBalancer, exposing all four BoJ ports externally. Defence-in-depth companion to PR#130 (Cowboy bind tightening). Together: BoJ binds loopback AND k8s Service is internal-only.Draft because this is breaking for anyone running the manifest as-is with a LoadBalancer in production. Owner gates merge on confirming no such deployment will break (header comment shows a kustomize/helm overlay for legacy/standalone needs).
All 18 CI checks green. Estate cross-check:
hypatia,rsr-certifier,opsm-serviceall use ClusterIP for backends — this PR brings BoJ in line. The only LoadBalancer in the estate issvalinn-gateway(a gateway, not a backend).Refs standards#100 / #91, no Closes.Phase E audit residue still open:
- Action chore(deps): bump the actions group with 2 updates #7:
stapeln.toml:100APP_HOST = "[::]"→ loopback (packager; separate concern). - Optional follow-up:
NetworkPolicyfor defence-in-depth beyond ClusterIP — not Phase E acceptance critical, can be filed as a tracked issue.
Ready to pick up #7 next.
🤖 Generated with Claude Code
- Action chore(deps): bump the actions group with 2 updates #7:
Fourth Phase E PR opened — boj-server#132 (audit action #7, container APP_HOST)
Filed as
boj-server#132: fix(container): APP_HOST defaults to 127.0.0.1 (HCG tier-2 E1 prereq) — DRAFT.Materialises audit action item #7. Scope expanded during implementation from one site (the audit named
stapeln.toml) to three sites —entrypoint.sh× 2 andcompose.prod.yamlhad the same[::]default that the audit missed. All three feed the same Zig-adapter--hostflag, so they need to flip together.Now the consumer-side audit is fully addressed across the three companion PRs:
Audit # Layer PR State #6 Elixir Cowboy bind boj-server#130 open, 18 checks green #8 k8s Service type boj-server#131 DRAFT, 18 checks green #7 Zig adapter APP_HOST defaults boj-server#132 DRAFT, CI didn't auto-trigger (see PR comment) — manual Governance dispatch green Defence in depth achieved: Elixir Cowboy binds loopback (#130) AND Zig adapter binds loopback (#132) AND k8s Service is internal-only (#131). Three independent layers, any one of which would mitigate the §3.4 invariant violation.
CI anomaly on #132:
pull_requestworkflows did not fire on the branch. Other PRs in this session (#130, #131) triggered normally. Likely cause: estate-wide CI concurrency-pool exhaustion from the active sweep122 campaign (Terminal-7 PID 244005). Manualworkflow_dispatchof Governance returned all 6 jobs green. Documented in PR#132 comment.Phase E audit-residue close-out: the only optional follow-up still open is a
NetworkPolicyfor defence-in-depth beyond ClusterIP. Not Phase E acceptance critical. Can file as a tracked issue if you want.Refs standards#100 / #91, no Closes.🤖 Generated with Claude Code
Optional follow-up filed — boj-server#135 (NetworkPolicy)
Filed
boj-server#135for the optional NetworkPolicy defence-in-depth layer noted in the consumer-side audit. Not Phase E acceptance critical; priority Low. Captures the proposed manifest shape, threat models it covers beyond ClusterIP, override path for non-HCG-fronted deployments, and CNI-support caveats. Picks up after Phase E exit unless an incident raises priority.Phase E audit residue is now fully tracked.
🤖 Generated with Claude Code
10 remaining items
- added 9 commits that reference this issue
on Jun 14, 2026 Next step (re-verified 2026-08-27): precisely scoped and untouched — flip
boj-serverTrustfile line 900tier_2_gateway.statusfromPENDING, and write the rollback runbook (docs/DEPLOYMENT.adocandSCOPED-DEPLOYMENT.adocexist; rollback does not).→ Gated on #91's artifacts actually passing — presence was verified, green was not. Run the mTLS/E2E/bench suites first; if green, this is small. The staging-validation half is the real cost.
- added a commit that references this issue
on Sep 22, 2026
Weeks 8–12. Wire tier-2 into BoJ; fill Trustfile.a2ml [CLOUDFLARE_EDGE_SECURITY].rate_limiting.tier_2_gateway PENDING; staging validation; rollout + rollback runbook. On landing: unblocks the unified core going public → unblocks K9 Dogfood green → idaptik#77 closable.
Parent: standards#91 (ADR-0004 phase E). Single-channel: do not work out of phase order; A→B→C→D→E.