The scale ladder: shared frame_sampling core + the UK national build consumer (#627)

[juaristi22]

S1 of the #627 scale-ladder lane, in two commits: the shared sampling core promoted out of the US stacked pipeline, then the UK national build consuming it. Together they deliver #627's ask — an identity-bound sample_fraction in the UK national build with US semantics, rungs f001/f010/f100, and "boring before billed" (#624) wiring.

Commit 1 — promote the sampling core into microcosm.build.frame_sampling

• The seeded whole-household sampler, mass renormalization, and receipt shape move out of us_runtime.stacked_spine (the engine_tables promotion-over-third-copy precedent). The US wrapper keeps its schema pin, pre-assembly guard, and exact error messages, and delegates the draw.
• Behavior preservation, proven: digests captured from the pre-promotion sampler are pinned in test_frame_sampling.py and reproduce exactly — same RNG stream, same receipt fields in the same order — through a UK-shaped schema, proving schema independence at the same time. The full US stacked-spine suite passes unchanged; receipt extensions (sampling_unit/strata/forced_unit_inclusions) appear only when the feature is exercised, so US checkpoint identities and manifests are untouched.
• Country policy enters as parameters: a sampling-unit key (a drawn unit brings all its household rows), per-unit strata with proportional monotone floors, and forced retentions unioned after the draw (RNG stream untouched, receipted).

Commit 2 — the UK consumer, shaped by what the compact actually is

The certified compact is a mid-pipeline artifact with baked-in structure, so the UK policy (uk_runtime/national_sampling.py):

• samples whole canonical clone families (clone reversal always lands on a surviving clone_index=0 row — frs_hmrc_leaves fence);
• force-retains the argmax-ID families so the re-derived clone multiplier keeps its digit count;
• stratifies by channel flags × the canonical row's region — the clone-0 quota cells vary by family region, so proportionality must hold within canonical region for the SPI rebuild's per-cell #base ≥ #dead quota (spi_support._sample_replacement_household_ids) to survive deterministically; a post-sample quota check fails closed, turning the uniform-clone-structure assumption into a runtime receipt;
• renormalizes sampled mass to the full-source total (kernel-minted MassChangeRecord) and refreshes the exported household_weight column in place, so the result satisfies validate_uk_national_frame.

Build seam: sampling runs after load_uk_national_frame and before provenance binding — the certified-candidate fence attests the frame the stages actually consume. At fraction=1.0 the sampler is never invoked: full-scale builds are structurally byte-invariant (asserted).

Driver: rung-validated --sample-fraction + separate --sample-seed (dev sweeps vary one draw at a time; --seed keeps the SPI replacement draw); fraction/seed/rung pinned into the checkpointed run_config as strings (exact-equality canonical JSON — two rungs on one checkpoint dir refuse instead of cross-resuming); the receipt rides the build record and stdout payload; and a sampled build refuses a canonical release id — rung artifacts are receipts, never releases. That interim fence stands until the gate battery's release_candidate wiring takes over at the #611 consumer half (the durable, country-agnostic home; no production caller passes it today in either country).

What this unblocks

• S2/S3 of the lane: local 1%/10% builds generating the National staging rebuild fails the current terminal gates: constant source_year release column and SPI weight tail above the reviewed maximum #630 adjudication evidence (the terminal-gate report is written before the failure raise, so receipts survive expected dev-scale gate failures).
• The S5 full-scale checkpointed run doubling as UK national staging onto the outer stage runtime, increment 3 of #612 #635's credentialed post-merge confirmation (Finish the #612 tails: drop the carrier household_weight column; narrow the crosswalk clone entry #654's operational tail).

Fences

• Track B (One country-agnostic gate battery: phased evaluation, unconditional reporting, per-country gate selection (#578) #611 inc 2) touches stacked_spine.py — land this first or rebase trivially. File-disjoint from the A1 lane (PR UK consumer half of the gate battery: uk/gates.json, bindings, and the contract doc (#611 increment 1) #655).
• The One country-agnostic gate battery: phased evaluation, unconditional reporting, per-country gate selection (#578) #611 consumer half (A2) rewires national_build.py's gate block; this PR's insertion sits upstream of it and does not touch the gate files.

Refs #627, #624, #630, #612, #654.

🤖 Generated with Claude Code

[juaristi22]

Adversarial review completed — three independent skeptical reviewers (shared-core behavior preservation; UK policy vs the real fences; driver/fences/receipts), every finding re-verified before acting. Fixes landed as 1778cc8.

Confirmed and fixed:

1. (real bug) A stale *.rung_abort.json survived a later successful build at the same staging path — now unlinked with the other sidecars, and the derived path is registered in the pairwise-distinct fence so it can never overwrite a read-only gate input.
2. The non-releasability regex was laxer than the microcosm-data contract it mirrors (fixed-width year, hardcoded tiers): now [1-9][0-9]* widths with the alternation built from the ratified UK_RELEASE_TIERS.
3. A programmatic checkpointed rung build could carry a sampling-blind run_config (cross-rung resume hazard on a fully-checkpointed rerun): the library now requires the sampling identity in run_config for fraction < 1.0.
4. (coverage hole — the drift class that broke the first credentialed rung run) The sampler was never tested against the real _resolve_candidate_lineage, and the sampling fixture violated the fence's person//1000 == household invariant. The fixture is now fence-valid and a new integration test requires sampled frames to resolve with the full frame's exact multiplier and person-level SPI/CG offsets.
5. frame_sampling hardening: integer-typed unit ids enforced (fractional ids collided in the selection digest), strata-only receipts emit the sampling_unit block instead of a requested_household_count that contradicts the declared exact-count rule, and the core reads person entity/membership from the schema.
6. Rung evidence stays a source fact: source_signal_rows/structural_zero are computed from the unrestricted raw survey surface, so a rung can never assert a structural zero from a sampled-away population.

Verified and accepted (recorded, not fixed):

• Per-stratum floors zero out small strata silently at tiny fractions — receipted in the strata block; immaterial at real UK region sizes.
• A rung cannot distinguish a compact genuinely missing raw people from sampling loss; the full-scale completeness fence stays strict and remains the guarantee.
• The terminal gate report/attestation does not carry the rung; the durable field belongs to the One country-agnostic gate battery: phased evaluation, unconditional reporting, per-country gate selection (#578) #611 schema-4 battery report at the UK migration (release_candidate + sampling block), not to schema-3 churn now.

Claims that survived attack (for reviewer confidence): the default draw is bit-for-bit the pre-refactor US draw (200-trial fuzz, byte-identical receipts); US manifests/identities unchanged; 900 seeded UK draws pass the real fence with exact reversal constants; the six forced argmax pins are sufficient; the per-cell SPI quota preservation is structural for pipeline-built compacts (regions are family-constant by construction); the in-place household_weight refresh cannot alias the loaded frame (Frame copies tables at construction); resume re-samples deterministically; str(error) in the abort receipt cannot carry unit-record values (the only reachable stratified split's y is sign codes).

🤖 Generated with Claude Code

[vahid-ahmadi]

The promotion is well done. Parameterising country policy (unit_ids, unit_strata, forced_unit_ids) instead of forking, and pinning the promoted US RNG stream with captured digests in _PINNED_DRAWS, is exactly the right way to move a sampler between pipelines — the pin is what makes "same seed, same draw" a claim rather than a hope. The two invariants in the module docstring (whole lineages, full-selection-is-a-no-op) are stated crisply and both are actually enforced in code.

uk_source_family_units is the part I'd want a second pair of eyes on, and it mostly holds up: re-deriving the fence's arithmetic and then forcing retention of the argmax-carrying families so the multiplier's digit count and both offsets stay stable is a genuinely clever solve. Refusing pre-draw on inputs the stage fence would reject is the right place for that check.

Notes, roughly in order of how much they matter.

1. sampled.table("household")["household_weight"] = ... assumes table() returns a live reference.

sampled.table("household")["household_weight"] = sampled.weights_for("household").values

If Frame.table() ever returns a copy (or a defensive view), this assignment succeeds and does nothing, and the exported column silently keeps its pre-normalisation values while the typed weights are correct. The comment explains why it's done in place (column position, and therefore staging payload column order), which is a good reason — but the correctness depends on an invariant of Frame that isn't visible here.

Worth either asserting it right after (assert np.allclose(sampled.table("household")["household_weight"], ...)) or exposing a Frame method that refreshes an exported column in place. This is the one place in the PR where a silent wrong answer is possible rather than a loud failure.

2. normalize_sampled_household_mass validates the divisor but not the target.

if not np.isfinite(sampled_mass) or sampled_mass <= 0.0:
    raise ValueError(...)
factor = float(target_mass / sampled_mass)

target_mass gets no such check, so a NaN or non-positive target produces a NaN/negative factor that propagates into the weights and the MassChange record. The caller currently passes receipt["incoming_household_mass"], which is trustworthy — but the function is exported and country-agnostic. Same guard on both sides would be cheap.

3. The floors-to-zero refusal ignores forced units.

requested_units < 1 raises before forced_unit_ids is considered, so a configuration with forced retention that would guarantee a non-empty sample still fails closed. For the UK path forced is always non-empty (six argmax pins), so at very small fractions on a small frame the error message — "sample fraction floors to zero source families" — will be true of the draw but not of the realised sample. Might be intended; if so a sentence in the docstring would save someone the confusion.

4. Dead check in _aligned_int_array.

if not np.issubdtype(array.dtype, np.integer):
    raise ...
...
if pd.isna(array).any():
    raise ValueError(f"{source_name} {label} must not contain missing values.")

By the time the second check runs the dtype is guaranteed integer, so pd.isna is always all-False. Harmless, but it reads as protection that isn't there. (The equivalent check in _int_column in national_sampling.py runs before the int cast, so it does bite — the asymmetry is what makes this one look intentional.)

5. Smaller things

• strata labels are built as f"region={region_by_household[int(unit)]}". If a region code ever arrives as 1 in one build and 1.0 in another, those are different strata with no error. _int_column-style coercion on the region column would remove the ambiguity.
• receipt["strata"] reports requested_units per group, but forced inclusions can push realized_units above it. That's honest given forced_unit_inclusions sits alongside, but a reader diffing the two blocks will trip over it once.
• The comment at frame_sampling.py:305-313 explaining why requested_household_count is omitted under strata (per-group floors need not sum to the global floor, so emitting it would contradict EXACT_COUNT_RULE) is excellent — that's the kind of thing that gets silently "fixed" back in six months. Good that it names the adversarial-review finding.

6. Process

This is still marked draft, and #658 is stacked on top of it (uk-source-year-exclusion-630 → uk-scale-ladder-627), so #658 is blocked until this lands. If the draft status is stale it'd be worth flipping it so the stack can move.

[juaristi22]

Thanks @vahid-ahmadi — all six points addressed in fe94809, in your order:

1. Live-reference assumption on the weight-column refresh — fixed, with one nuance. A silent wrong answer was actually already impossible two calls later: validate_uk_national_frame(sampled) runs after the refresh and does an exact np.array_equal between the exported column and the typed weights, so a copy-returning Frame.table would have failed the build loudly. But that guarantee was non-local (a refactor dropping or reordering the revalidation would silently reintroduce the gap), so the sampler now verifies the assignment persisted right at the assignment site and fails closed naming the actual cause. There's a regression test that monkeypatches Frame.table to return copies and asserts the refusal fires.

2. target_mass unguarded — fixed. Same finite-and-positive guard on both sides of the ratio, with a test over nan/inf/0/negative.

3. Floors-to-zero ignoring forced units — intended, now documented. Forced units are identity pins (they exist to keep the fence's reversal constants derivable), not a sample; a fraction whose proportional draw floors to zero has no representative content regardless of what forced retention would realize, so it stays fail-closed. The docstring now says exactly that.

4. Dead pd.isna check — removed. You read it right: unreachable after the integer-dtype guard. The asymmetry with _int_column (whose check runs pre-cast and does bite) stands.

5a. Region label ambiguity — fixed, but not with int coercion. I checked the real compact: region is string-coded (LONDON, SCOTLAND, …), so _int_column-style coercion would refuse every real build. Instead the sampler now enforces non-empty string labels fail-closed — a numeric region code is refused outright, which removes the 1 vs 1.0 two-strata ambiguity entirely rather than normalizing it. Test covers the refusal.

5b. requested_units vs realized_units — fixed. Each stratum entry now carries added_beyond_draw, so realized = requested + added_beyond_draw reads directly in place, and a test asserts that identity per group plus its sum matching the global forced_unit_inclusions block.

6. Process — the PR is out of draft, and #658 is rebased onto the new head.

[vahid-ahmadi]

fe94809 addresses all six. Verified each against the diff rather than the summary:

1. Weight refresh — the local np.array_equal check right at the assignment site, with an error naming Frame.table as the cause. test_weight_column_refresh_verifies_the_live_reference monkeypatches Frame.table to return copies and asserts the refusal fires; suppressing the pandas ChainedAssignmentError with a comment saying the warning is the failure mode is a nice touch. Your point that validate_uk_national_frame already made a silent wrong answer impossible is fair — the value here is that the failure is now local and named, and survives a refactor that reorders the revalidation.
2. target_mass — same finite-and-positive guard on both sides.
3. Floors-to-zero — accepted, and the docstring reason (forced units are identity pins, not sample content) is the right justification. Good that it's written down rather than left as behaviour.
4. Dead pd.isna — gone.
   5a. Region labels — _str_column enforcing non-empty strings is a better answer than the int coercion I suggested. Refusing a numeric region code outright removes the 1/1.0 ambiguity instead of papering over it, and checking the real compact first was the right instinct.
   5b. added_beyond_draw per stratum — realized = requested + added_beyond_draw now reads in place, with the per-group and global-sum identity both asserted.
5. Out of draft, #658 rebased.

No new findings. This looks ready to me.