Skip to content

UK enablement tooling: source-inventory de-hardcode, UK consumer-artifact suite, legacy ETL cleanup #135

Description

@juaristi22

Lanes: ordinary maintainer tooling under db/; if the suite wiring touches policyengine_chronicle/** consumer surfaces, that PR runs under ledger-contract-maintainer (.github/chronicle-agents.yml)
Related: the academic_year period type landed via #131/#136; the wave packages this depended on are all merged (#137/#139/#141/#155/#159/#163, plus the sex-split facts in #173)

Revised 2026-08-13 for the #166 ruling (selection contracts live in the consumer; Chronicle is facts-only): PR 2 no longer embeds target profiles — the UK consumer artifact is a facts-only feed, and everything profile-shaped now lives with the contracts in microcosm (microcosm#707/#708) or retires under #172. PRs 1 and 3 are unaffected; PR 3 is now unblocked.

Three small, independent enablers for the UK target migration.

PR 1 — de-hardcode db/pe_source_inventory.py

DEFAULT_PE_UK_ROOT = Path("/Users/maxghenis/PolicyEngine/policyengine-uk-data") (and the US twin) key the UK ingestion inventory to one person's checkout. Replace with env vars / CLI arguments with no default, and fail with a clear message when unset. The inventory's UK rows (ons 13, hmrc 6, dwp 5, obr 3, voa 2, nrs 2, nomis 2, slc 1, nts 1) were the working checklist for waves 1–3; the waves are done, but the inventory remains the supersession cross-check for PR 3 and for any backfill lane.

PR 2 — UK consumer-artifact suite (facts-only)

Add the UK suite wiring so chronicle build-suite / build-consumer-artifact can produce a hash-pinned facts-only UK bundle covering the wave 1–3 packages: consumer_facts.jsonl + manifest hashes, no embedded profiles. This is the feed microcosm#622 consumes (ledger_facts_sha256-pinned) to compile the UK contract (uk/uk_national_targets.json, microcosm#707) into a target registry — per the #166 ruling the artifact carries facts and provenance, and nothing selection-shaped. Per-profile embedding and coverage.json are being retired from the artifact under #172; do not build UK wiring against them.

Known behavior to design around (observed on the US side, June 2026): a single---year bundle build drops year-pinned sources; the working pattern is per-suite build at each pinned year + concatenated consumer_facts.jsonl. Either fix the year handling properly here or codify the per-suite+concat pattern for the UK suite — pick one and document it.

PR 3 — retire the legacy hardcoded UK ETLs (unblocked)

db/etl_obr.py and db/etl_ons.py are hardcoded March-2024 dicts feeding the legacy three-table schema. docs/agent-source-package-harness.md already declares the old ETL loaders "compatibility and migration inputs, not the preferred agent" lane. The #133 OBR/ONS packages are merged, so this is ready: delete these two modules (or reduce them to explicit tombstones pointing at the packages), so no future agent mistakes them for the ingestion path. Verify the wave-2 packages supersede every value they carry before deletion.

Acceptance

  • Inventory runs against arbitrary checkout paths; CI proves the no-default failure mode.
  • One documented UK suite command produces a hash-pinned, facts-only UK consumer artifact (no profile surfaces referenced).
  • Legacy UK ETLs gone or tombstoned, with supersession verified in the PR description.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions