UK wave 2: housing base facts — PIPR bulletin rents and EHS weekly housing costs (#133) - #159
UK wave 2: housing base facts — PIPR bulletin rents and EHS weekly housing costs (#133)#159juaristi22 wants to merge 5 commits into
Conversation
…using costs (PolicyEngine#133) The publisher rates behind uk-data's three housing/* rate-x-count targets (the products stay excluded per the facts-only ADR): - ons-pipr-private-rent-march-2026 (5 facts, html_tables_and_text): the archived March-2026 bulletin's average monthly private rent - UK 1,374 / England 1,430 / Wales 828 / Scotland 1,022 at month 2026-02, and NI 875 at month 2025-12, the bulletin's own NI keying. The 17.8MB monthly workbook is a named exclusion: its ~1.9M-cell xlsx_used_range parse per bundle build is operationally prohibitive; values cross-checked against its Table 1 before authoring. - mhclg-ehs-weekly-housing-costs-2023-24 (304 facts, ods_used_range, new mhclg source): EHS 2023-24 rented-sectors Annex Table 1.7 whole-table - mean+median weekly costs x 5 tenures x 6 characteristic blocks + all-households, England, fiscal_year 2023. Publisher 'u' suppression and the mortgagor-ineligibility dashes are carried as record-set splits, not skipped cells. All 309 built facts re-derived independently against the lane parsers' canonical grids; R2 storage published (credentialed publish-raw) before commit so every fact carries a real raw_r2_uri. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…olicyEngine#133) 44,668 facts / 90 packages / 36 sources / 85 tables / 99 periods (+month:2026-02, +month:2025-12, +fiscal_year:2023 keys; mhclg joins by_source; country:K02000001 +1 = the PIPR UK fact). All six pinned structures regenerated from one measured build, pins sweep-verified. Also updates the source checklist: PolicyEngine#141 sections retitled as merged, the garbled Stat-Xplore heading restored, housing moved to ported with its two unresolved factors documented. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…yEngine#133) The per-wave 'Ported' sections were legacy from the porting sequence; a single table now carries every national target family with explicit status and PR columns. Families whose publisher source cannot be obtained are first-class rows marked blocked (SPP Review NI-relief bases: source lost; housing mortgage payment: mismatched citation, no source; UK-wide tenure counts: no pinned publisher table) instead of being bundled into other rows or trailing sections. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…base (PolicyEngine#133) 44,867 facts / 96 packages / 37 sources / 91 tables / 116 periods — post-PolicyEngine#155 main (44,558) plus this branch's 309 housing facts. All six pinned structures regenerated from one measured build; pin sweep clean (country:K02000001 3,819 -> 3,820, the PIPR UK fact). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
96c54c0 to
8b31a74
Compare
|
Restacked on the post-#155 main and re-measured: 44,867 facts / 96 packages / 37 sources / 116 periods (= #155's 44,558 + this branch's 309), full local suite green before push. This head also replaces the wave-era checklist layout: |
vahid-ahmadi
left a comment
There was a problem hiding this comment.
The two things I'd most want from a wave-closing PR are both here: the counts reconcile, and the things that couldn't be sourced are written down as first-class blocked rows rather than quietly omitted.
Arithmetic checks out — 44,558 → 44,867 is +309, splitting exactly as mhclg 304 + ons +5 (5,112 → 5,117), with source_count 36 → 37 for the new mhclg source and source_package_count 94 → 96. Geographies unchanged at 1,068 is consistent with PIPR landing on countries that already exist.
The mortgage-citation finding deserves to outlive this PR. uk-data cites the ONS rent bulletin for a £1,100/month mortgage payment, and the bulletin carries no mortgage figure at all. Refusing to port a number that has no source, and naming the nearest sourced anchor (AT1.7 all-mortgagors mean, £222.42/week ≈ £964/month) instead of quietly substituting it, is exactly right. But right now that discovery lives in a PR body, and PR bodies stop being read the moment they merge. It's a live mis-citation in a downstream repo — worth an issue on uk-data so total_mortgage gets rebuilt on a real source rather than inheriting a citation that doesn't support it.
Two things I'd change or think about.
1. The £0.02/week artifacts are carried as ordinary facts, with the explanation only in a YAML comment.
The header comment says it plainly:
mortgagor costs include nominal GBP 0.02/week values (note 5), which is why the 75-or-over mortgagor median is 0.023
But that comment lives in the source package, which the consumer of the bundle never sees. Downstream, median weekly housing cost, mortgagors, HRP 75+ is 0.023 with nothing on the fact marking it a publisher artifact — and 2p/week is the kind of number that looks like a unit error, so the charitable reader treats it as a bug and the incurious one divides by it.
Contrast the handling of the u disclosure cells and the missing housing-support mortgagor columns: those become record-set splits, so the measures are simply absent. Absence is self-describing. A present-but-artifactual value is the case that needs the marker, and it's the one that doesn't have one. If there's a per-fact slot for a publisher caveat, note 5 belongs in it; if there isn't, that's worth knowing, because this won't be the last table with one.
Related: note 2's italic small-sample flags are dropped as "formatting-only". Italics are formatting, but small sample is a data property — worth confirming the flag isn't recoverable from a sample-size column you're already excluding as survey metadata.
2. Provisionality has no representation on the assertion axis.
Your reasoning on the February 2026 PIPR value is right as far as the axis goes: a provisional estimate of an elapsed period is an observation, not a projection, so assertion correctly stays at the default. But that leaves the axis unable to say the one thing that matters about it — ONS will revise this number, and after they do, the bundle carries a stale value indistinguishable from a final one.
#154 gave us observation vs source_projection. This is the first package to want a third state (or an orthogonal provisional flag), and the gap is structural rather than something to solve here. Given how well #157 worked out as a filed issue from a review, this feels like the same shape — especially since a consumer resolving "latest" against a revisable figure is exactly the silent-wrong-answer case the assertion axis was built to prevent.
Smaller
- Choosing the archived bulletin over the 47,800-row workbook is well justified with measured numbers (>2 minutes and gigabytes of RSS on every bundle build is not a tax worth paying for 5 facts), and cross-checking all five cell-by-cell against the workbook before authoring is the right diligence. Worth confirming the bulletin URL is a permanent archive link rather than a live path — the
expected_valueguards carry full prose ("This is £47 (3.5%) higher than 12 months ago"), which is a strong guard but fails on any re-publication that rewords. - Keying NI to
month:2025-12because the bulletin itself does, rather than forcing it onto the GB month or dropping it, is the honest call and worth the comment it has. - The checklist consolidation into one full-inventory table with blocked rows as first-class entries is a real improvement over per-wave "Ported" sections — those were only ever meaningful during the porting.
- This is the sixth wave PR to hand-edit the whole-bundle constants. #157 is still open; no action for this PR, just noting the cost is still being paid.
Closes the last unported wave-2 national family on #133: the publisher base facts behind uk-data's three
housing/*targets. All three uk-data targets are rate×count products and stay excluded (computed) per the facts-only ADR — what lands here are the publisher rates they multiply, plus the honest paper trail for the factors that have no source.Packages (2 packages, 309 facts, registry-only)
ons-pipr-private-rent-march-2026(5 facts,html_tables_and_text)The archived March-2026 "Private rent and house prices, UK" bulletin — the exact citation in uk-data's
housing.py. Average monthly private rent: UK £1,374, England £1,430, Wales £828, Scotland £1,022 atmonth:2026-02, and Northern Ireland £875 as its own record set atmonth:2025-12— NI's series lags the GB collection and the bulletin itself keys NI to December, so it is period-keyed truthfully rather than dropped or mis-keyed. February 2026 is a provisional PIPR estimate (bulletin wording) subject to routine revision; it is an observation of an elapsed period, soassertionstays at the default.Why the bulletin and not the dataset: the monthly price-statistics workbook (25-March-2026 edition) is a 47,800-row × 40-column table whose
xlsx_used_rangeparse materializes ~1.9M source cells on every bundle build (>2 minutes, gigabytes of RSS — measured before authoring). Regional and property-size averages are therefore a named exclusion pending a row-filtered workbook lane; LA grain is wave-3 scope (#134). All five quoted values were cross-checked cell-by-cell against that workbook's Table 1 before authoring.mhclg-ehs-weekly-housing-costs-2023-24(304 facts,ods_used_range)Annex Table 1.7 (mean and median weekly housing costs, £/week) of the EHS 2023-24 rented-sectors Chapter 1 annex — uk-data's citation for social rent. Whole-table port: six characteristic blocks (age of HRP, economic status of HRP, region, weekly income quintile, receipt of housing support, dependent children) plus the all-households row, × five tenure columns × {mean, median}. New
mhclgsource. The all-households row carries the three headline anchors:rent_social) — exact sourcePublisher markers are carried as record-set structure, not skipped cells: the full-time-education row's mortgagor cells are
u(disclosure control, note 1) and the housing-support rows have no mortgagor columns (mortgagors are ineligible, note 3), so those two blocks declare the eight non-mortgagor measures only. Known quirks kept as-is and guarded: mortgagor costs include nominal £0.02/week values (note 5) — that is why the 75-or-over mortgagor median is 0.023; sample-size columns are survey metadata (named exclusion).What deliberately does not land, and why
~UK-wide with no table citation; EHS is England-only, so no single publisher table carries them. Constructing UK-wide counts means composing EHS headline AT1.1 with three devolved housing surveys plus an adjudicated composition rule — its own curation with visible adjudications, tracked on the checklist, not smuggled in here.total_mortgagedeclaration will need either a new adjudicated source (e.g., UK Finance data, if accessible) or to be rebuilt on the EHS mean.Verification
document_numbersrows); every value matches, including full-precision floats behind the published roundings (118.12004… → 118).£ per week (mean)/(median)), the table title, the source footer, publisher suppression markers, and — for the bulletin — element index, element type, and full paragraph text per fact.docs/pe-uk-source-checklist.mdconsolidated into one full-inventory table (the per-wave "Ported" sections were porting-sequence legacy): every national family in a single table withstatusandPRcolumns, and source-not-obtainable families as first-class blocked rows (SPP Review, the mortgage payment, the UK-wide tenure counts) instead of notes bundled into other rows. The previously garbled Stat-Xplore heading is gone with the sections.chronicle publish-rawrun for both artifacts; manifests commit realstorage:blocks (R2 round-trip verified).🤖 Generated with Claude Code