UK consumer half of the gate battery: uk/gates.json, bindings, and the contract doc (#611 increment 1) - #655
UK consumer half of the gate battery: uk/gates.json, bindings, and the contract doc (#611 increment 1)#655juaristi22 wants to merge 9 commits into
Conversation
Adversarial reviewI attacked the PR's own claims — especially "behaviour-preserving by construction" and "thresholds-as-declared-data" — with executable probes against the branch. Three claims survived; three did not fully survive. Findings ranked by severity; fixes follow in a subsequent commit. FindingsF1 — MEDIUM-HIGH (correctness, A2-blocking): empty F2 — MEDIUM (declared-data integrity, migration-window): the export-surface exclusion register merges rather than replaces. Verified: F3 — MEDIUM (test coverage): the terminal branch of the coverage binding never executes in any test. The differential feeds the fixture evaluator to both sides (justified — the real gate needs the committed licensed manifest), and the preflight tests cover only F4 — LOW (consistency): extra keys inside a zero-weight declaration are silently ignored. The binding constructs F5 — TRIVIAL (diagnostics): the unknown-key refusal doesn't name the offending entry. Claims that survived attack
VerdictApprove-with-changes: F1 fixed in-PR (the equivalence claim is otherwise overbroad), F2 notes + F3 test cheap in-PR, F4/F5 opportunistic. None of the findings invalidates the declared spec itself — 🤖 Generated with Claude Code |
Review resolutions (826beba)
All affected suites green locally (125 tests) + repo-wide ruff clean; CI running. 🤖 Generated with Claude Code |
vahid-ahmadi
left a comment
There was a problem hiding this comment.
Declaring the battery as data with the bindings kept separate from the verdicts is the right decomposition, and the docstring in battery_bindings.py is unusually clear about what a binding may and may not do ("a binding adapts evidence, it never re-implements a verdict"). The detail that the two legacy evaluators get their names re-minted while any other unexpected name still fails closed is a nice piece of care — it would have been much easier to just relax the name check.
The notes fields in gates.json are doing real work too; "an absent audit is not a passing audit" on uk_weights_audit is worth more than the parameter block above it.
1. The closed-world check stops one level short of where its own rationale points.
_GATE_ENTRY_KEYS = frozenset(
{"id", "gate", "phase", "criticality", "parameters", "not_applicable", "notes"}
)with the comment:
an unknown key is refused rather than ignored, because a silently dropped key (a typo'd
parameters, a speculative extension) would run the gate on defaults while the declared intent vanished frompolicy_sha256— an unattested threshold.
That argument is exactly right, and it applies just as forcefully inside parameters, which isn't validated at all. A gate entry with minimum_ess_fration: 0.01 passes from_mapping, ships inside policy_sha256, and runs weight_ess on its default threshold — the precise failure the comment describes, one level deeper. Since each gate name already implies its parameter vocabulary, having the binding declare its accepted parameter keys and validating against that at load would close it.
2. Two release-blocking gates are declared with their thresholds still in code.
uk_input_mass_parity and uk_qrf_tail_concentration are both criticality: release_blocking, but their notes say thresholds "stay runtime-armed pending microcosm#630". So the declared policy hash covers the reference identity and the register resource names, while the numbers that decide pass/fail live outside it. That's a documented interim state rather than an oversight, but it does mean policy_sha256 currently under-describes those two gates — worth making sure #630's follow-up is tracked against this file specifically, since the notes are the only thing recording the debt.
3. Exact-measurement fences with no headroom.
"maximum_max_to_median_ratio": 1151.2542195939373The note is explicit that this is the certified June measurement and that re-baselining is a reviewed parameter edit, never a loosen-to-pass — good discipline, and I'm not arguing for headroom. But if the comparison is <= against a value recomputed from the same data, the gate sits exactly on the boundary, and any float-level difference (BLAS version, summation order, platform) flips it to failing with no signal distinguishing "drift" from "last bit differs". Same question for the two maximum_zero_weight_rows: 100000 caps, which equal the exact shipped counts. Worth confirming the comparison direction makes bit-identical rebuilds safe.
4. A note here goes stale as soon as #658 lands.
uk_degenerate_release_surface says:
The microcosm#630 source-year finding lands here as a reviewed exclusion once its register entry is approved.
#658 is open and does exactly that, committing degenerate_reviewed_exclusions.json and moving the policy digest. Whichever of these merges second should update the other's text — and since #658 changes country_package.json resources while this PR adds gates.json to the same package, they'll likely conflict there too.
5. Minor
not_applicableis in the allowed key set but unused by any UK gate. Fine if the vocabulary is shared across countries; just flagging that nothing in this PR exercises it.- The
allowed_extra_columnslist (46 entries) and thereviewed_exclusionsmap are both reviewed registers now living inline ingates.json, while the input-mass and QRF-tail registers are separate resources referenced by name. Two conventions for the same concept in one file — worth a line in the contract doc saying which to reach for. - The policy string mentions "A delivered-take-up gate is deferred until the shared gate vocabulary grows a take-up name" — is that tracked in an issue? It's the one gate that's absent rather than declared, so it's the one a reader can't see.
Review point 1 on PR #655: the entry-level closed-world check stopped one level short of its own rationale. Inside `parameters`, most evaluators splat unknown keys into the gate call — failing closed, but mid-battery as a TypeError — while the two selective-read preflight evaluators silently ignored extras, letting a declared key ship inside policy_sha256 while governing nothing. Every binding now declares the parameter vocabulary it can route (`parameter_keys`, fail-closed empty by default, mirroring each gate function's full keyword surface so microcosm#630 still lands as pure parameter edits), and `validate_gate_parameters` checks the whole manifest when the battery is armed — at GateBatteryRun construction and in evaluate_phase — refusing any key outside the vocabulary with the entry named, before a single gate runs. Entries whose gate has no binding keep their named evidence_absent gap. The contract doc gains the rule plus the register-placement convention (loader-owned registers are package resources; plain reviewed lists live inline). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vahid-ahmadi
left a comment
There was a problem hiding this comment.
47e5c1d closes my first point properly, and further than I asked.
validate_gate_parameters runs over the whole manifest, from both evaluate_phase and GateBatteryRun.__init__, and is exported so a consumer arming a battery by hand can pre-validate. parameter_keys defaulting to frozenset() — a binding that declares nothing accepts nothing — is the right direction to fail. The error message reusing the original argument ("would sit inside policy_sha256 while governing nothing") keeps the rationale attached to the failure.
Two things I checked specifically because they're where this could have gone wrong:
- Every UK binding declares its vocabulary, and
test_every_declared_uk_parameter_is_inside_its_binding_vocabularypins the committeduk/gates.jsonagainstUK_GATE_REGISTRY. Without that, the fail-closed default would have refused the entire UK manifest at arm time. Good that it exists. - Skipping unbound gates is justified in the docstring, and
test_an_unbound_gate_keeps_its_named_gap_whatever_it_declaresconfirmsmacro_realismstill lands onevidence_absent. Worth noting the consequence out loud, since it's the one soft edge left: a parameter typo'd on a gate that has no binding yet ships insidepolicy_sha256unchallenged, and only starts failing the day someone writes the binding. That's the correct trade — refusing would block declaring gates ahead of their implementations — but it means the vocabulary check is weakest exactly where the spec is most speculative.
The relative_tolerence test is a good choice of fixture; that's the realistic typo.
One stale note to update before merge. The uk_degenerate_release_surface entry still says:
The microcosm#630 source-year finding lands here as a reviewed exclusion once its register entry is approved.
#658 has now landed that register (and upgraded all three registers to schema-2 approval receipts with expiry under #610). Since #658 also touches country_package.json resources and this PR adds gates.json to the same list, the two will conflict there regardless — worth resolving the note in the same pass.
My other two points from the first review stand as questions rather than blockers: the two release-blocking gates whose thresholds are still runtime-armed (so policy_sha256 under-describes them), and whether the exact-measurement fences (1151.2542195939373, the two 100000 caps) are compared in a direction that keeps a bit-identical rebuild passing. Both are fine to answer in a comment rather than a change.
|
Thanks — taking point 1 as a change, with one correction to its mechanism for the record; the rest answered inline. 1. Taken, with a correction to the failure story. A typo'd
As of 47e5c1d, every binding declares its parameter vocabulary ( 2. Agreed — documented interim state, now tracked where it survives note edits. Left a comment on microcosm#630 pointing at the two entries ( 3. Verified: every fence is inclusive, so bit-identical rebuilds are safe. 4. Yes — sequencing is ours. Whichever of #655/#658 merges second updates the other's 5. |
vahid-ahmadi
left a comment
There was a problem hiding this comment.
Correcting the record on my point 1: you're right about the mechanism. The splatting evaluators pass **dict(parameters) into the gate function, so a typo'd minimum_ess_fration raises TypeError and _evaluate_gate turns that into a failed-closed gate. My claim that weight_ess would run on its default threshold was wrong for those paths — the battery did fail, just badly.
The structural argument holding on the preflight side is the part that mattered, and you've identified it more precisely than I did: parameters.get("check") and a source_coverage that ignores parameters entirely are where an extra key genuinely shipped inside policy_sha256 while governing nothing. Fail-closed-by-splat being a per-evaluator convention rather than a guarantee is the right way to state the remaining risk.
Thanks for the fence audit with line numbers — ratio > maximum, fraction < minimum, count > maximum_zero_weight_rows, abs(error) <= max_abs. Equality passing everywhere is what I wanted to confirm: a rebuild reproducing the certified measurement sits on the passing side, and an upward float difference failing is the declared policy working rather than flakiness. That closes my point 3.
Point 2 tracked on #630, point 5 answered — and the inline-vs-resource rule you landed on ("a register an existing runtime loader owns is a named package resource; a plain reviewed list with no loader lives inline, where editing it moves the policy hash directly") is a better articulation than the question deserved. Worth keeping in the contract doc exactly as phrased.
Only open item from me is the stale uk_degenerate_release_surface note, which is now squarely a sequencing matter — #658 has landed the register and retargeted to main, so whichever of the two merges second takes that edit and the country_package.json resolution.
A typo'd or speculative key in a gate entry was silently ignored: the gate ran on its defaults while the declared intent vanished from policy_sha256 — an unattested threshold, exactly the failure class the battery exists to prevent (microcosm#611). Entry validation is now closed-world over the documented key vocabulary, with a refusal test. BE's gates.json already conforms, so no consumer changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The certified June UK battery becomes a declared country spec: two preflight entries (coverage-manifest currency, required build stages) plus the eleven terminal gates, phases ["preflight", "terminal"]. Thresholds move from module constants into entry parameters at their exact certified values — ESS fraction, the max-to-median weight ratio acceptance fence, target-fit error, the two reviewed June zero-weight strata, and the export-surface registers. Per the microcosm#327 rule, the input-mass parity reference arrives as a declared input (identity + canonical digest; the totals stay uncommitted under the UKDS licence), and both reviewed-exclusion registers are referenced as package resources. Input-mass and QRF-tail tolerances stay runtime-armed pending the microcosm#630 adjudication, whose receipted values land here as parameter edits. Gate names use the country-neutral shared vocabulary; ids stay UK-flavored. TestUKGatesManifest pins every declared parameter against the legacy module constant so spec and code cannot drift during the migration window. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
uk_runtime/battery_bindings.py supplies the GateBinding implementations behind every gate uk/gates.json declares: the shared DEFAULT_REGISTRY plus UK bindings that adapt the Frame onto the legacy duck-attr evidence surface, construct reviewed strata from declared parameters, and pass undeclared parameters through so an unknown key fails closed. The two legacy result names (uk_release_input_coverage, qrf_tail_concentration) are re-minted under the declared neutral names; any other unexpected name still fails the executor's name check. The input-mass and QRF-tail overrides derive candidate evidence from the frame and hold the supplied reference and exclusion registers to the spec-declared pins. The coverage binding serves both declared entries, running frameless in its preflight manifest-currency mode. Gate comparisons stay in terminal_gates/weighted_integrity unchanged; the registry is consumed via GateBatteryRun(registry=...) when the national build migrates onto the battery executor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dicts test_uk_battery_bindings.py pins the behaviour-preservation contract: evaluate_phase over uk/gates.json with UK_GATE_REGISTRY reproduces uk_terminal_gate_report gate for gate over identical synthetic evidence — same verdicts, same failure lines, same details — both fully armed and with seeded defects (weight blowout, unreviewed export column, bad target fit, unweighted fit). Both sides are built from one fixture function so evidence asymmetry cannot masquerade as a behavioural difference. The chartered semantic delta is asserted positively, not papered over: where the legacy report omits unevidenced gates under its sealed membership contract, the battery records every declared entry as evidence_absent with the missing keys named, blocking release candidates only. Also covered: the BE-style empty-evidence plumbing run over the UK spec, preflight bindings against the committed coverage manifest, frozen-parameter construction of the reviewed strata, re-minting confined to the declared legacy names, and fail-closed behaviour for unknown parameters and a drifted declared reference pin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docs/gate-battery-contract.md is the consumer contract for the country-agnostic battery: the anatomy of gates.json and what each of the three declaration digests covers, the evidence contract with the canonical artifact-key vocabulary, the five-status outcome taxonomy and the two-axis blocking rule, the BE/UK/US-pool postures side by side, the microcosm#327 reference rule with the UK input-mass entry as the worked case, and the onboarding checklist enumerating every input a new country supplies — including the boxed warning that the live UK release still signs with the POPULACE_-prefixed key while the battery mints the MICROCOSM_ name, an owner-held rotation at the contract flip. The worked example is a test, not an examples/ script (examples/ sits outside pytest's testpaths and would rot unexercised): test_gate_battery_contract_example.py runs a minimal synthetic country on the shared registry alone — one passing gate, one failing gate, one named evidence gap — through write-then-block, and the doc narrates it. The microcosm-build README points at the contract from its gate list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-binding test F1: the UK registry now overrides weights_audit — a fit stage that ran but emitted an empty record set is a failed audit (the legacy guard), not the vacuous pass the shared binding would produce; the differential gains the present-but-empty scenario, failure-text identical on both sides. F2: the export-surface entry notes, the contract doc, and the input-mass binding comment now state the migration-window mechanism plainly — the exclusion register is merged with the legacy constant and the declared reference identity is inert at runtime, with the spec-pin tests holding declared and enforced equal until the constants retire and the bindings flip to replace-semantics. F3: the coverage binding's terminal branch now runs the real gate in a test (stub engine, minimal manifest) and is asserted equal to the direct call, re-minted onto the neutral name. F4: unknown keys inside a zero-weight declaration are refused rather than silently ignored, matching the entry-level closed-world stance. F5: the unknown-entry-key refusal names the offending entry id. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The doc was half reference, half status report: issue citations, migration-window caveats, a signing-key transition box, a roadmap section, and a planned-country column. That stage-tracking content already lives where point-in-time state belongs — the plan, the issues, the PR discussion, the gates.json notes, and code comments — so the doc now explains only how the battery works and how a country uses it: the three disciplines, the gates.json anatomy and its digests, the evidence contract and binding vocabulary, outcomes and blocking, the narrated worked example, the BE/UK postures as they stand, the reference rule, and the onboarding checklist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review point 1 on PR #655: the entry-level closed-world check stopped one level short of its own rationale. Inside `parameters`, most evaluators splat unknown keys into the gate call — failing closed, but mid-battery as a TypeError — while the two selective-read preflight evaluators silently ignored extras, letting a declared key ship inside policy_sha256 while governing nothing. Every binding now declares the parameter vocabulary it can route (`parameter_keys`, fail-closed empty by default, mirroring each gate function's full keyword surface so microcosm#630 still lands as pure parameter edits), and `validate_gate_parameters` checks the whole manifest when the battery is armed — at GateBatteryRun construction and in evaluate_phase — refusing any key outside the vocabulary with the entry named, before a single gate runs. Entries whose gate has no binding keep their named evidence_absent gap. The contract doc gains the rule plus the register-placement convention (loader-owned registers are package resources; plain reviewed lists live inline). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebasing onto main surfaced a real divergence the differential tests caught: #658 made the committed schema-2 register the terminal report's default degenerate policy, so the legacy path ran the policy of record while the battery binding ran bare — dormant/expired state differed (household.source_year dormant in one, absent in the other). The binding now follows the spec's own idiom for the other two exclusion gates: uk/gates.json declares reviewed_exclusions_resource, the evaluator pins the declared name against the runtime's register constant, and the gate runs uk_default_degenerate_reviewed_exclusions() — the exact policy of record the legacy report resolves for None. The entry's note is updated: the anticipated #630 approval landed via #658. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
47e5c1d to
20c9062
Compare
What this is
The UK consumer half of #611 increment 1 (the tooling half merged in #619 with zero production consumers). The certified June UK battery becomes declared data plus a binding registry, the contract is documented with a runnable worked example, and the national build is deliberately untouched — the orchestration swap onto
GateBatteryRunis the next increment.Five commits, each green:
country_spec.py): an unknowngates.jsonentry key is refused rather than silently dropped outsidepolicy_sha256— a typo'dparameterswould otherwise run the gate on defaults while the declared threshold went unattested.uk/gates.json: phases["preflight", "terminal"]; 2 preflight entries (coverage-manifest currency, required build stages) + the 11 terminal gates, allrelease_blocking, country-neutralgatenames with UK-flavoredids. Thresholds move from module constants into entryparametersat their exact certified values (ESS fraction, the max-to-median weight-ratio acceptance fence, target-fit error, the two reviewed June zero-weight strata, the export-surface registers). Per the Export input-mass-parity gate references the raw base, flagging correct calibration gains on PUF-imputed columns #327 rule, the input-mass parity reference is a declared input (identity + canonical digest; the licensed totals stay uncommitted) and both reviewed-exclusion registers are named as package resources. Input-mass and QRF-tail tolerances stay runtime-armed pending the National staging rebuild fails the current terminal gates: constant source_year release column and SPI weight tail above the reviewed maximum #630 adjudication, which lands here as parameter edits.TestUKGatesManifestpins every declared parameter against the legacy module constant so spec and code cannot drift during the migration window.uk_runtime/battery_bindings.py:UK_GATE_REGISTRY= the shared defaults plus UK bindings that adapt theFrameonto the legacy duck-attr evidence surface, construct reviewed policy objects from frozen declared parameters, pass undeclared parameters through so an unknown key fails closed, re-mint exactly the two legacy result names (uk_release_input_coverage,qrf_tail_concentration) onto the shared vocabulary, and hold the runtime-supplied reference/registers to the spec-declared pins. The coverage binding serves both declared entries, frameless in its preflight mode.test_uk_battery_bindings.py):evaluate_phaseover the spec reproducesuk_terminal_gate_reportverdict for verdict on identical synthetic evidence — fully armed and with seeded defects — with both sides built from one fixture function so evidence asymmetry cannot masquerade as a behavioural difference. The chartered semantic delta is asserted positively: the legacy report omits unevidenced gates; the battery records namedevidence_absentgaps that block release candidates only. Plus the BE-style empty-evidence plumbing run, preflight bindings against the committed manifest, and fail-closed unit tests (unknown parameter, drifted declared pin, impostor result name).docs/gate-battery-contract.md— anatomy ofgates.jsonand what each declaration digest covers, the evidence contract with the canonical artifact-key table, the outcome taxonomy and two-axis blocking rule, BE/UK/US-pool postures side by side, the Export input-mass-parity gate references the raw base, flagging correct calibration gains on PUF-imputed columns #327 reference rule generalized, and the onboarding checklist of every input a new country supplies (the doc is a timeless reference; migration-window caveats such as thePOPULACE_UK_→MICROCOSM_UK_signing-key transition live in the PR discussion, gates.json notes, and code comments — rotation stays owner-held at the flip). The worked example is a CI-run test (test_gate_battery_contract_example.py), narrated by the doc.Review notes
release_input_coveragegate name with a{"check": "manifest_current"}parameter (the registry is keyed by gate name;requires_frameis parameter-dependent by design). If that reads poorly, the alternative is a newALLOWED_GATE_FUNCTIONSname — flagged rather than done, since the vocabulary is owner-held.not_applicableentries: the only candidate (delivered take-up) has no allowlist gate name yet; recorded as a named gap in the manifestpolicyprose and the doc's roadmap.gates.jsonmovesCountrySpec.fingerprintfor the UK; the only pin was the resource tuple intest_country_spec.py, updated in the same commit. No other consumer loads the UK spec.Closes nothing on its own; advances #611 (increment 1, consumer half). Refs #578, #327, #630.
🤖 Generated with Claude Code