The scale ladder: shared frame_sampling core + the UK national build consumer (#627)

changelog.d/627-frame-sampling-core.added.md

@@ -0,0 +1 @@
+`microcosm.build.frame_sampling`: the scale-ladder sampling core promoted out of `us_runtime.stacked_spine` (#627). One seeded whole-unit household sampler for every country's #624 rungs — the US stacked draw is reproduced bit-for-bit (same RNG stream, same receipts), and country policy arrives as parameters: an optional sampling-unit key (whole clone families), per-unit strata with proportional floors, and receipted forced retentions added after the draw.

changelog.d/627-uk-scale-ladder.added.md

@@ -0,0 +1 @@
+The UK national build gains the #624 scale ladder (#627): `--sample-fraction` (0.01 smoke / 0.10 dev / 1.0 full) with a separate `--sample-seed`, sampling the loaded certified compact at canonical-clone-family grain — families whole, argmax-ID families force-retained so the clone multiplier is stable, the draw stratified by channel flags × canonical region so the SPI rebuild quota survives per cell (re-asserted post-sample, fail-closed), and sampled mass renormalized to the full-source total with a kernel-minted record. Fraction and seed are pinned into the checkpointed run config (two rungs on one checkpoint directory refuse instead of cross-resuming), the rung receipt rides the build record, and a sampled build refuses a canonical release id — rung artifacts are receipts, never releases. At 1.0 the sampler is never invoked; full-scale builds are structurally unchanged.

changelog.d/657-rung-abort-receipt.added.md

@@ -0,0 +1 @@
+Rung builds abort with a receipt on the one named dev-scale statistical edge (#657, closed without pipeline changes): when the SPI imputation's stratified split refuses a singleton class on a sub-1.0 build, the driver writes a `*.rung_abort.json` receipt naming the edge, the rung, and the re-roll remedy, and exits with a distinct code. The computation is never altered, and only this named edge is receipted — unknown exceptions crash loudly.

packages/microcosm-build/src/microcosm/build/uk_runtime/frs_hmrc_leaves.py

@@ -174,6 +174,10 @@ class UKFRSHMRCRetainedLeavesResult:
     structural_zero_columns: tuple[str, ...]
     input_content_identity: str
     output_content_identity: str
+    #: Raw-survey people outside the candidate base. Zero on a full-scale
+    #: build (the completeness fence raises otherwise); on a #627 rung
+    #: sample it receipts how much of the raw surface the rung dropped.
+    source_people_outside_candidate: int = 0
 
     def evidence(self) -> dict[str, object]:
         """Return aggregate, JSON-safe evidence for a national build driver."""
@@ -197,6 +201,9 @@ def evidence(self) -> dict[str, object]:
                 "capital_gains_person_id_offset": (self.capital_gains_person_id_offset),
                 "raw_source_people": self.raw_source_people,
                 "candidate_people": self.candidate_people,
+                "source_people_outside_candidate": (
+                    self.source_people_outside_candidate
+                ),
             },
             "retained_leaves": {
                 column: {
@@ -215,6 +222,9 @@ class UKFRSHMRCRetainedLeavesStageTransform:
 
     adult_tab_path: Path
     benefits_tab_path: Path
+    #: Declared #627 rung build: relaxes the raw-surface completeness fence
+    #: into a receipted count. Never set on a release build.
+    sampled_rung: bool = False
     last_result: UKFRSHMRCRetainedLeavesResult | None = field(
         default=None,
         init=False,
@@ -224,13 +234,16 @@ class UKFRSHMRCRetainedLeavesStageTransform:
     def from_raw_frs_directory(
         cls,
         raw_frs_directory: str | Path,
+        *,
+        sampled_rung: bool = False,
     ) -> UKFRSHMRCRetainedLeavesStageTransform:
         """Resolve the two permitted tables from a CLI-supplied directory."""
 
         directory = Path(raw_frs_directory).expanduser()
         return cls(
             adult_tab_path=directory / "adult.tab",
             benefits_tab_path=directory / "benefits.tab",
+            sampled_rung=sampled_rung,
         )
 
     def __call__(self, frame: Frame) -> Frame:
@@ -242,6 +255,7 @@ def __call__(self, frame: Frame) -> Frame:
             frame,
             adult_tab_path=self.adult_tab_path,
             benefits_tab_path=self.benefits_tab_path,
+            sampled_rung=self.sampled_rung,
         )
         return self.last_result.frame
 
@@ -345,8 +359,17 @@ def retain_uk_frs_hmrc_leaves(
     *,
     adult_tab_path: str | Path,
     benefits_tab_path: str | Path,
+    sampled_rung: bool = False,
 ) -> UKFRSHMRCRetainedLeavesResult:
-    """Read two raw FRS tables and retain the adjudicated HMRC constituents."""
+    """Read two raw FRS tables and retain the adjudicated HMRC constituents.
+
+    ``sampled_rung`` declares a #627 scale-ladder build: the candidate base
+    deliberately carries only a sampled subset of source families, so the
+    completeness fence (every raw-survey person present in the base) cannot
+    hold. The raw surface is restricted to surviving canonicals and the
+    dropped count is receipted instead — never silently. Full-scale builds
+    keep the strict fence.
+    """
 
     validate_uk_national_frame(frame)
     input_content_identity = uk_frame_content_identity(frame)
@@ -372,11 +395,24 @@ def retain_uk_frs_hmrc_leaves(
     unknown_source_ids = sorted(
         set(source_leaves.index) - lineage.canonical_raw_person_ids
     )
-    if unknown_source_ids:
+    if unknown_source_ids and not sampled_rung:
         raise ValueError(
             "Raw FRS retained leaves contain person identity value(s) absent "
             f"from the certified candidate base: {unknown_source_ids[:5]}."
         )
+    source_people_outside_candidate = len(unknown_source_ids)
+    # Signal-row evidence stays a fact about the SOURCE at every rung:
+    # structural_zero must never be asserted from a sampled-away surface
+    # (adversarial-review finding). The rung also cannot distinguish a
+    # compact genuinely missing raw people from sampling loss — that check
+    # remains the full-scale fence's, which stays strict.
+    full_source_leaves = source_leaves
+    if unknown_source_ids:
+        # A rung sample deliberately drops most source families; restrict the
+        # raw surface to the surviving canonicals and receipt the count.
+        source_leaves = source_leaves.loc[
+            source_leaves.index.isin(list(lineage.canonical_raw_person_ids))
+        ]
 
     person = frame.table("person").copy()
     aligned = source_leaves.reindex(lineage.source_person_ids, fill_value=0.0)
@@ -407,7 +443,7 @@ def retain_uk_frs_hmrc_leaves(
         source_leaves=source_leaves,
     )
     source_signal_rows = {
-        column: int((source_leaves[column] > 0.0).sum())
+        column: int((full_source_leaves[column] > 0.0).sum())
         for column in FRS_HMRC_RETAINED_LEAF_COLUMNS
     }
     structural_zero_columns = tuple(
@@ -424,6 +460,7 @@ def retain_uk_frs_hmrc_leaves(
         capital_gains_person_id_offset=lineage.capital_gains_person_id_offset,
         raw_source_people=len(source_leaves),
         candidate_people=len(person),
+        source_people_outside_candidate=source_people_outside_candidate,
         source_signal_rows=source_signal_rows,
         structural_zero_columns=structural_zero_columns,
         input_content_identity=input_content_identity,

packages/microcosm-build/src/microcosm/build/uk_runtime/hmrc_restoration.py

@@ -212,6 +212,9 @@ class UKHMRCIncomeStageTransform:
     qrf_estimators: int = 100
     donor_sample_size: int | None = DEFAULT_SPI_DONOR_SAMPLE_SIZE
     spi_prior_mass_share: float = DEFAULT_SPI_PRIOR_MASS_SHARE
+    #: Declared #627 rung build: the mid-stage effective-mass floor defers
+    #: to the terminal input-coverage gate. Never set on a release build.
+    sampled_rung: bool = False
     last_result: UKHMRCIncomeRestorationResult | None = field(
         default=None,
         init=False,
@@ -398,6 +401,7 @@ def __call__(self, frame: Frame) -> Frame:
             qrf_estimators=self.qrf_estimators,
             donor_sample_size=self.donor_sample_size,
             spi_prior_mass_share=self.spi_prior_mass_share,
+            sampled_rung=self.sampled_rung,
         )
         return self.last_result.frame
 
@@ -502,8 +506,18 @@ def restore_uk_hmrc_income_family(
     qrf_estimators: int = 100,
     donor_sample_size: int | None = DEFAULT_SPI_DONOR_SAMPLE_SIZE,
     spi_prior_mass_share: float = DEFAULT_SPI_PRIOR_MASS_SHARE,
+    sampled_rung: bool = False,
 ) -> UKHMRCIncomeRestorationResult:
-    """Run the admissible real-donor replay without biased calibration."""
+    """Run the admissible real-donor replay without biased calibration.
+
+    ``sampled_rung`` declares a #627 scale-ladder build: sparse imputed
+    columns can legitimately restore near-zero effective mass on a small
+    sample, so the mid-stage effective-mass floor defers to the terminal
+    input-coverage gate — which evaluates the same surface and records a
+    receipted verdict — instead of aborting the build. The per-column
+    shares reach the replay report either way. Full-scale builds keep the
+    strict raise.
+    """
 
     assert_uk_hmrc_income_source_contract_current()
     _validate_certified_candidate_identity(certified_candidate)
@@ -567,7 +581,7 @@ def restore_uk_hmrc_income_family(
         for name, share in distributional_mass_shares.items()
         if share < DEFAULT_MINIMUM_NONDEFAULT_MASS_SHARE
     }
-    if insufficient:
+    if insufficient and not sampled_rung:
         raise RuntimeError(
             "Rebuilt SPI channel did not restore required effective-mass "
             f"coverage: {insufficient}."

packages/microcosm-build/src/microcosm/build/uk_runtime/national_build.py

@@ -21,6 +21,10 @@
 
 import microcosm.build.uk_runtime.national_frame as _national_frame
 import microcosm.build.uk_runtime.release_input_coverage as _release_input_coverage
+from microcosm.build.frame_sampling import (
+    validate_sample_fraction,
+    validate_sample_seed,
+)
 from microcosm.build.gates import GateReport, GateResult
 from microcosm.build.uk_runtime.national_frame import (
     UKStagingProvenance,
@@ -29,6 +33,10 @@
     uk_time_period,
     validate_uk_national_frame,
 )
+from microcosm.build.uk_runtime.national_sampling import (
+    UK_SAMPLE_SEED_DEFAULT,
+    sample_uk_national_frame,
+)
 from microcosm.build.uk_runtime.release_input_coverage import (
     PolicyEngineUKCoverageEngine,
     assert_uk_release_input_coverage_build_stages,
@@ -144,6 +152,8 @@ class UKNationalBuildResult:
     stage_names: tuple[str, ...]
     terminal_gates: GateReport
     terminal_gate_path: Path
+    #: The #627 rung receipt; ``None`` on a full-scale (fraction 1.0) build.
+    sampling_receipt: Mapping[str, object] | None = None
 
     @property
     def input_coverage(self) -> GateResult:
@@ -323,6 +333,8 @@ def build_uk_national_dataset(
     input_coverage_path: str | Path | None = None,
     checkpoint_dir: str | Path | None = None,
     run_config: Mapping[str, object] | None = None,
+    sample_fraction: float = 1.0,
+    sample_seed: int = UK_SAMPLE_SEED_DEFAULT,
 ) -> UKNationalBuildResult:
     """Run ordered national stages, hard-gate the result, and stage an H5.
 
@@ -335,6 +347,16 @@ def build_uk_national_dataset(
     configuration is refused by the runtime, and an unpinned resume is
     exactly the drift hazard checkpoints exist to prevent, so a checkpointed
     build without a ``run_config`` is refused here.
+
+    ``sample_fraction`` below 1.0 is the #627 scale ladder: the loaded frame
+    is sampled at clone-family grain (see
+    :func:`~microcosm.build.uk_runtime.national_sampling.sample_uk_national_frame`)
+    before provenance binding, so the certified-candidate fence attests the
+    frame the stages actually consume. At 1.0 the sampler is never invoked —
+    full-scale builds are structurally byte-invariant to it. A checkpointed
+    sampled run must carry the fraction and seed inside ``run_config`` (the
+    driver does); otherwise two rungs pointed at one checkpoint directory
+    would silently resume across each other.
     """
 
     requested_input_path = Path(input_h5).expanduser()
@@ -387,7 +409,26 @@ def build_uk_national_dataset(
             "a checkpointed UK national build requires run_config: the "
             "content-addressed run identity is what makes a resume safe."
         )
+    validate_sample_fraction(sample_fraction, label="UK sample")
+    validate_sample_seed(sample_seed, label="UK sample")
+    if (
+        checkpoint_dir is not None
+        and sample_fraction != 1.0
+        and "sampling" not in run_config
+    ):
+        raise ValueError(
+            "a checkpointed rung build requires the sampling identity inside "
+            "run_config: two rungs pointed at one checkpoint directory must "
+            "refuse, never cross-resume."
+        )
     frame, provenance = load_uk_national_frame(requested_input_path)
+    sampling_receipt: Mapping[str, object] | None = None
+    if sample_fraction != 1.0:
+        # Sample before provenance binding: the fence attests the sampled
+        # frame, and the stages never learn a rung existed.
+        frame, sampling_receipt = sample_uk_national_frame(
+            frame, fraction=sample_fraction, seed=sample_seed
+        )
     # Stages whose fences bind the loaded bytes (the SPI stage's
     # certified-candidate check) receive the load provenance and the loaded
     # frame explicitly — provenance travels beside the frame, never inside
@@ -449,6 +490,7 @@ def build_uk_national_dataset(
         stage_names=tuple(stage.name for stage in materialized_stages),
         terminal_gates=terminal_gates,
         terminal_gate_path=diagnostic_path,
+        sampling_receipt=sampling_receipt,
     )
 
 
@@ -637,7 +679,9 @@ def _mass_log_from_stored(value: object) -> tuple[MassChangeRecord, ...]:
                 )
             )
         except (KeyError, TypeError, ValueError) as exc:
-            raise ValueError("Stored UK microcosm mass-log entry is malformed.") from exc
+            raise ValueError(
+                "Stored UK microcosm mass-log entry is malformed."
+            ) from exc
     return tuple(records)