Skip to content

UK wave 2: housing base facts — PIPR bulletin rents and EHS weekly housing costs (#133) - #159

Open
juaristi22 wants to merge 5 commits into
PolicyEngine:mainfrom
juaristi22:uk-wave2-housing-base-facts
Open

UK wave 2: housing base facts — PIPR bulletin rents and EHS weekly housing costs (#133)#159
juaristi22 wants to merge 5 commits into
PolicyEngine:mainfrom
juaristi22:uk-wave2-housing-base-facts

Conversation

@juaristi22

@juaristi22 juaristi22 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Closes the last unported wave-2 national family on #133: the publisher base facts behind uk-data's three housing/* targets. All three uk-data targets are rate×count products and stay excluded (computed) per the facts-only ADR — what lands here are the publisher rates they multiply, plus the honest paper trail for the factors that have no source.

Packages (2 packages, 309 facts, registry-only)

ons-pipr-private-rent-march-2026 (5 facts, html_tables_and_text)
The archived March-2026 "Private rent and house prices, UK" bulletin — the exact citation in uk-data's housing.py. Average monthly private rent: UK £1,374, England £1,430, Wales £828, Scotland £1,022 at month:2026-02, and Northern Ireland £875 as its own record set at month:2025-12 — NI's series lags the GB collection and the bulletin itself keys NI to December, so it is period-keyed truthfully rather than dropped or mis-keyed. February 2026 is a provisional PIPR estimate (bulletin wording) subject to routine revision; it is an observation of an elapsed period, so assertion stays at the default.

Why the bulletin and not the dataset: the monthly price-statistics workbook (25-March-2026 edition) is a 47,800-row × 40-column table whose xlsx_used_range parse materializes ~1.9M source cells on every bundle build (>2 minutes, gigabytes of RSS — measured before authoring). Regional and property-size averages are therefore a named exclusion pending a row-filtered workbook lane; LA grain is wave-3 scope (#134). All five quoted values were cross-checked cell-by-cell against that workbook's Table 1 before authoring.

mhclg-ehs-weekly-housing-costs-2023-24 (304 facts, ods_used_range)
Annex Table 1.7 (mean and median weekly housing costs, £/week) of the EHS 2023-24 rented-sectors Chapter 1 annex — uk-data's citation for social rent. Whole-table port: six characteristic blocks (age of HRP, economic status of HRP, region, weekly income quintile, receipt of housing support, dependent children) plus the all-households row, × five tenure columns × {mean, median}. New mhclg source. The all-households row carries the three headline anchors:

tenure mean £/week uk-data's figure
all social renters 118.12 £118/week (rent_social) — exact source
private renters 236.59 — (England PRS complement to the UK-level PIPR rate)
all mortgagors 222.42 (~£964/month) see mortgage note below

Publisher markers are carried as record-set structure, not skipped cells: the full-time-education row's mortgagor cells are u (disclosure control, note 1) and the housing-support rows have no mortgagor columns (mortgagors are ineligible, note 3), so those two blocks declare the eight non-mortgagor measures only. Known quirks kept as-is and guarded: mortgagor costs include nominal £0.02/week values (note 5) — that is why the 75-or-over mortgagor median is 0.023; sample-size columns are survey metadata (named exclusion).

What deliberately does not land, and why

  • The tenure counts (7.5m mortgagors / 5.4m private renters / 5.0m social renters). uk-data marks all three ~ UK-wide with no table citation; EHS is England-only, so no single publisher table carries them. Constructing UK-wide counts means composing EHS headline AT1.1 with three devolved housing surveys plus an adjudicated composition rule — its own curation with visible adjudications, tracked on the checklist, not smuggled in here.
  • The £1,100/month mortgage payment has no source. uk-data cites the ONS rent bulletin for it, which carries no mortgage figure (mismatched citation). Nothing to port; the nearest sourced anchor is AT1.7's all-mortgagors mean (£222.42/week ≈ £964/month), which lands here. The populace-side total_mortgage declaration will need either a new adjudicated source (e.g., UK Finance data, if accessible) or to be rebuilt on the EHS mean.

Verification

  • Both packages' 309 built facts re-derived independently against the lane parsers' canonical grids (the EHS ODS grid and the bulletin's document_numbers rows); every value matches, including full-precision floats behind the published roundings (118.12004… → 118).
  • Guard cells pin block headers, unit annotations (£ per week (mean)/(median)), the table title, the source footer, publisher suppression markers, and — for the bulletin — element index, element type, and full paragraph text per fact.
  • docs/pe-uk-source-checklist.md consolidated into one full-inventory table (the per-wave "Ported" sections were porting-sequence legacy): every national family in a single table with status and PR columns, and source-not-obtainable families as first-class blocked rows (SPP Review, the mortgage payment, the UK-wide tenure counts) instead of notes bundled into other rows. The previously garbled Stat-Xplore heading is gone with the sections.
  • Credentialed chronicle publish-raw run for both artifacts; manifests commit real storage: blocks (R2 round-trip verified).
  • Full bundle-expectation surface re-measured from one build; full local suite green before push.

🤖 Generated with Claude Code

juaristi22 and others added 5 commits August 12, 2026 11:29
…using costs (PolicyEngine#133)

The publisher rates behind uk-data's three housing/* rate-x-count targets
(the products stay excluded per the facts-only ADR):

- ons-pipr-private-rent-march-2026 (5 facts, html_tables_and_text): the
  archived March-2026 bulletin's average monthly private rent - UK 1,374 /
  England 1,430 / Wales 828 / Scotland 1,022 at month 2026-02, and NI 875
  at month 2025-12, the bulletin's own NI keying. The 17.8MB monthly
  workbook is a named exclusion: its ~1.9M-cell xlsx_used_range parse per
  bundle build is operationally prohibitive; values cross-checked against
  its Table 1 before authoring.
- mhclg-ehs-weekly-housing-costs-2023-24 (304 facts, ods_used_range, new
  mhclg source): EHS 2023-24 rented-sectors Annex Table 1.7 whole-table -
  mean+median weekly costs x 5 tenures x 6 characteristic blocks +
  all-households, England, fiscal_year 2023. Publisher 'u' suppression and
  the mortgagor-ineligibility dashes are carried as record-set splits, not
  skipped cells.

All 309 built facts re-derived independently against the lane parsers'
canonical grids; R2 storage published (credentialed publish-raw) before
commit so every fact carries a real raw_r2_uri.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…olicyEngine#133)

44,668 facts / 90 packages / 36 sources / 85 tables / 99 periods
(+month:2026-02, +month:2025-12, +fiscal_year:2023 keys; mhclg joins
by_source; country:K02000001 +1 = the PIPR UK fact). All six pinned
structures regenerated from one measured build, pins sweep-verified.
Also updates the source checklist: PolicyEngine#141 sections retitled as merged,
the garbled Stat-Xplore heading restored, housing moved to ported with
its two unresolved factors documented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…yEngine#133)

The per-wave 'Ported' sections were legacy from the porting sequence; a
single table now carries every national target family with explicit
status and PR columns. Families whose publisher source cannot be
obtained are first-class rows marked blocked (SPP Review NI-relief
bases: source lost; housing mortgage payment: mismatched citation, no
source; UK-wide tenure counts: no pinned publisher table) instead of
being bundled into other rows or trailing sections.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…base (PolicyEngine#133)

44,867 facts / 96 packages / 37 sources / 91 tables / 116 periods —
post-PolicyEngine#155 main (44,558) plus this branch's 309 housing facts. All six
pinned structures regenerated from one measured build; pin sweep clean
(country:K02000001 3,819 -> 3,820, the PIPR UK fact).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@juaristi22
juaristi22 force-pushed the uk-wave2-housing-base-facts branch from 96c54c0 to 8b31a74 Compare August 12, 2026 09:55
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Restacked on the post-#155 main and re-measured: 44,867 facts / 96 packages / 37 sources / 116 periods (= #155's 44,558 + this branch's 309), full local suite green before push.

This head also replaces the wave-era checklist layout: docs/pe-uk-source-checklist.md now carries one full-inventory table for every national target family, with explicit status and PR columns instead of per-wave "Ported" sections. Families whose publisher source cannot be obtained are first-class rows marked blocked — SPP Review NI-relief bases (source lost), the £1,100/month mortgage payment (mismatched citation, no source), and the UK-wide tenure counts (no pinned publisher table) — rather than being bundled into other rows' notes.

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two things I'd most want from a wave-closing PR are both here: the counts reconcile, and the things that couldn't be sourced are written down as first-class blocked rows rather than quietly omitted.

Arithmetic checks out — 44,558 → 44,867 is +309, splitting exactly as mhclg 304 + ons +5 (5,112 → 5,117), with source_count 36 → 37 for the new mhclg source and source_package_count 94 → 96. Geographies unchanged at 1,068 is consistent with PIPR landing on countries that already exist.

The mortgage-citation finding deserves to outlive this PR. uk-data cites the ONS rent bulletin for a £1,100/month mortgage payment, and the bulletin carries no mortgage figure at all. Refusing to port a number that has no source, and naming the nearest sourced anchor (AT1.7 all-mortgagors mean, £222.42/week ≈ £964/month) instead of quietly substituting it, is exactly right. But right now that discovery lives in a PR body, and PR bodies stop being read the moment they merge. It's a live mis-citation in a downstream repo — worth an issue on uk-data so total_mortgage gets rebuilt on a real source rather than inheriting a citation that doesn't support it.

Two things I'd change or think about.

1. The £0.02/week artifacts are carried as ordinary facts, with the explanation only in a YAML comment.

The header comment says it plainly:

mortgagor costs include nominal GBP 0.02/week values (note 5), which is why the 75-or-over mortgagor median is 0.023

But that comment lives in the source package, which the consumer of the bundle never sees. Downstream, median weekly housing cost, mortgagors, HRP 75+ is 0.023 with nothing on the fact marking it a publisher artifact — and 2p/week is the kind of number that looks like a unit error, so the charitable reader treats it as a bug and the incurious one divides by it.

Contrast the handling of the u disclosure cells and the missing housing-support mortgagor columns: those become record-set splits, so the measures are simply absent. Absence is self-describing. A present-but-artifactual value is the case that needs the marker, and it's the one that doesn't have one. If there's a per-fact slot for a publisher caveat, note 5 belongs in it; if there isn't, that's worth knowing, because this won't be the last table with one.

Related: note 2's italic small-sample flags are dropped as "formatting-only". Italics are formatting, but small sample is a data property — worth confirming the flag isn't recoverable from a sample-size column you're already excluding as survey metadata.

2. Provisionality has no representation on the assertion axis.

Your reasoning on the February 2026 PIPR value is right as far as the axis goes: a provisional estimate of an elapsed period is an observation, not a projection, so assertion correctly stays at the default. But that leaves the axis unable to say the one thing that matters about it — ONS will revise this number, and after they do, the bundle carries a stale value indistinguishable from a final one.

#154 gave us observation vs source_projection. This is the first package to want a third state (or an orthogonal provisional flag), and the gap is structural rather than something to solve here. Given how well #157 worked out as a filed issue from a review, this feels like the same shape — especially since a consumer resolving "latest" against a revisable figure is exactly the silent-wrong-answer case the assertion axis was built to prevent.

Smaller

  • Choosing the archived bulletin over the 47,800-row workbook is well justified with measured numbers (>2 minutes and gigabytes of RSS on every bundle build is not a tax worth paying for 5 facts), and cross-checking all five cell-by-cell against the workbook before authoring is the right diligence. Worth confirming the bulletin URL is a permanent archive link rather than a live path — the expected_value guards carry full prose ("This is £47 (3.5%) higher than 12 months ago"), which is a strong guard but fails on any re-publication that rewords.
  • Keying NI to month:2025-12 because the bulletin itself does, rather than forcing it onto the GB month or dropping it, is the honest call and worth the comment it has.
  • The checklist consolidation into one full-inventory table with blocked rows as first-class entries is a real improvement over per-wave "Ported" sections — those were only ever meaningful during the porting.
  • This is the sixth wave PR to hand-edit the whole-bundle constants. #157 is still open; no action for this PR, just noting the cost is still being paid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants