Add Laurens, IA — the first rural split (recall 0.390) and register it as the 8th US split - #152
Add Laurens, IA — the first rural split (recall 0.390) and register it as the 8th US split#152jonfroehlich wants to merge 4 commits into
Conversation
Phase 0 intent: rural / small-town streetscape on a consumer 360 rig. Every existing split is urban, suburban, or a college town; Laurens is 1.91 km2 and 1,264 people. It answers whether the operating point holds where sidewalks are intermittent and driveway aprons outnumber ramps. Imagery: Mapillary, GoPro Max 5760x2880, all captured 2025-11 by a single uploader in 19 sequences — the most uniform vintage of any split. Detection run: 4,495 panos, 9.3% with an operational detection. Strata are 5 top / 64 random / 25 empty = 94, not the standard 5/95/25. The shortfall is geometric, not a sampling artifact: _spread walks the whole candidate pool greedily, and the town saturates at 64 detection panos held 30 m apart. The unbiased column (random + empty) is therefore 89 panos against the usual 120, so its intervals are correspondingly wider. Strata definitions and the de-clustering rule are unchanged. 111 operational detections to adjudicate. Archive reconciled 1:1, zero decay. Still outstanding per docs/adding_a_benchmark_city.md: GT review, the BENCHMARK_SPLITS allowlist, and the downstream pooled numbers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SF2s8wWUM6HphV1xuYju21
digest 7485886ea08f671b over 94 panos, verify clean. Written now rather than after review so it records the bytes the reviewer actually sees — Mapillary re-serves thumb_original_url from a signed, expiring URL, so "the same pano id" is not a guarantee of the same pixels later. docs/adding_a_benchmark_city.md still does not mention this step; it postdates the runbook. Every other split carries one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SF2s8wWUM6HphV1xuYju21
94/94 panos reviewed, all 111 detections judged, 190 missed-ramp marks. Precision 0.898 (95% CI 0.827-0.942) unbiased 0.863 Recall 0.390 (95% CI 0.331-0.451) unbiased 0.325 Precision is unremarkable — mid-pack. Recall is roughly half the next-worst split (clovis 0.650), and three things say it is a real failure mode rather than a small or ramp-poor sample: - Laurens is ramp-RICH: 2.65 ramps/pano, 3rd of 10 splits, ahead of richmond, annapolis and morgantown. Misses/pano is 1.62 against a 0.45-1.19 range everywhere else. - The misses are NOT far-field. In all nine other splits missed ramps sit nearer the horizon than detected ones (delta -0.013 to -0.030 in normalized y); Laurens is the only split where the delta is positive (+0.004). It is missing near, well-resolved ramps. - Not a lighting artifact either. The reviewer noted leaf litter on three panos, but measured over a 2%-width window the missed and detected ramps sit in the same light: median luminance 94.6 vs 97.4, 47% vs 42% in shadow. The leaf/shadow reading is not supported. What the miss crops do show, visually and so far unmeasured, is flush or minimal-reveal ramps — a street cross-section close to at-grade, which is the rural design this split was added to probe. The GSV arm of this same town (runs/laurens_gsv, 2,137 panos, 2024-09, leaf-free) is the discriminator and is not yet reviewed: if its recall is also ~0.35 the cause is ramp geometry, if ~0.75 it is rig or season. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SF2s8wWUM6HphV1xuYju21
Phase 4-5 of docs/adding_a_benchmark_city.md. Extraction + gate: operating_point_curve.py extract --cities laurens (94 panos, local GPU) low_floor_sweep.py parity --cities laurens -> 111/111, 100.0% EXACT, med R 0.000. Bit-exact, which is what a Mapillary split should be. Registries: US_SPLITS in low_floor_sweep and miss_decomposition, TIER (mapillary), BENCHMARK_SPLITS, SERIES, and SPLIT_HEADER in scoreboard_render (a touchpoint the Phase 5 table does not list). tier_of() already resolves GoPro Max to action-modern, so no new branch. SERIES takes slot 7 (#4a3aa7) -- the categorical slot the manual_gold fold vacated, so no new hue is invented. Pooled numbers move, as the runbook warns they will: thr 0.30 P 0.9002 -> 0.8991 R 0.7927 -> 0.7640 (-2.9 pts) thr 0.55 P 0.9637 -> 0.9594 R 0.7223 -> 0.6864 (-3.6 pts) n_panos 859 -> 953 Precision is untouched; the recall drop is Laurens' own 0.390 entering the pool, not a regression anywhere else. farfield_forensics.py and silent_activation.py both read the live US_SPLITS but are FROZEN studies -- their tests pin the committed result JSONs to numbers quoted in the issues' section 0c. Registering a city silently restated both. Each now carries its own PUBLISHED_SPLITS / US_SPLITS tuple with the reason, and their tests read that instead, so the published findings stay reproducible and a future split cannot rewrite them by accident. Laurens is absent from those two studies for that reason alone; it is pooled everywhere else. Also refreshed: op/*.csv (sweep, hist, gtbias, floor, distance), op_cache/laurens.json, scoreboard.json, model_scoreboard.md and the five scoreboard figures. The split date range test moves to 2026-08-31, Laurens' review date. NOT resolved here -- eight tests in test_scoreboard.py still fail, and they should not be silenced by bumping constants. No challenger model has been scored on Laurens, so every one of them becomes partial coverage (7/8) and drops out of the pooled tables, and test_ap_ordering_is_not_an_artifact_of_the_floor now compares RampNet's 8-split AP against a 7-split YOLO AP -- 0.829 vs 0.730, a margin of 0.0993 against an asserted 0.1. That is a population mismatch, not a stale constant. See the PR for the decision it needs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SF2s8wWUM6HphV1xuYju21
Deep reviewRe-ran the branch head ( Two things changed underneath this PR since it was opened, and the review has to be read against them. PR #153 ( 1. High —
|
| line | prose says | the table beside it says |
|---|---|---|
| 3 | "Eighteen model legs, ten splits" | eleven splits |
| 7–12 | "YOLO11l … 0.604; Gemini 3.1 Pro … 0.608 … a dead heat" | neither model is in the headline table any more (both 7/8, partial) |
| 13–14 | "Its F1 spans 0.80–0.85 across the seven cities, a range of 0.053 … the only strong model that is also stable" | F1 range column: 0.54–0.85 |
| 18–19 | "0.969, above RampNet's 0.958 … against RampNet's 0.728" | RampNet 0.951 / 0.686 |
| 24 | "RampNet is the top score in all ten splits" | eleven; and after #153 it is false on laurens_mapillary (y26_pano 0.574 > 0.543) |
| 26–27 | "clovis … is the worst pooled city for 5 of the 12 models" | RampNet's worst is now laurens |
| 34 | "PR curves, pooled over the seven US splits" | eight |
| 42–44 | "The pool is seven cities, not ten. … The three held-out splits" | eight of eleven; three held-out becomes four after #153 |
The "stability" finding (line 13) is the one that matters scientifically: the sentence is now refuted by its own table, and a reader who trusts the prose gets the pre-Laurens picture.
#153 does not fix this — its model_scoreboard.md diff touches only lines inside the GENERATED markers. Prose edits outside those markers will merge cleanly; write them as "eight US splits" rather than naming laurens wherever possible so the downstream rename does not have to touch them.
4. Medium — the two operating-point figures were not regenerated
plot_operating_point.py gained the laurens SERIES entry and collect() reads ALL_SPLITS, so a regeneration would draw the eighth city — but docs/figures/operating_point_pr.png and docs/figures/storage_floor_ceiling.png are not in the diff (last touched bdd7d55, 2026-08-01). The committed figure is a seven-split plot sitting next to eight-split CSVs. The runbook's Phase 4 checklist has "Both figures regenerated and visually inspected" precisely because "an eighth series changes the legend layout and can collide with the curves". The body's "five figures regenerated" are the scoreboard figures, which are a different script.
#153 does not regenerate them either. Regenerate both here and look at them; #153 will need to run the same two scripts once more after its rename so the legend reads laurens_mapillary and includes the dashed laurens_gsv line it added to HELD_DASH.
5. Medium — the frozen-study fix was applied to two studies; three more have the same exposure
The PR's reasoning is right: a committed study whose script reads the live US_SPLITS is silently restated by every new registration. It was applied to farfield_forensics and silent_activation because their tests failed. But four more committed studies default --cities to the live tuple, and three of them do not even record which splits they covered:
| committed result | script default | records its split list? |
|---|---|---|
analysis_out/miss_decomposition.json |
US_SPLITS (now 8) |
no |
analysis_out/miss_taxonomy.json |
US_SPLITS (now 8) |
no |
analysis_out/silent_witness.json |
US_SPLITS (now 8) |
no |
analysis_out/fp_taxonomy.json |
US_SPLITS (now 8) |
yes — cities: the seven |
A default re-run of any of the first three now overwrites a seven-split result with an eight-split one, and nothing in the payload says which it is. None was re-run for this PR (they are unchanged in the diff), and the body does not say so — per CLAUDE.md, an unstated gap is indistinguishable from a hidden one. The numbers those files feed (docs/operating_point.md's "Pooled over the seven US splits" at lines 212, 265, 409; the #46 taxonomy rates) are therefore still seven-split figures under an eight-split registry.
Fix, minimal: have miss_decomposition.py, miss_taxonomy.py and silent_witness.py write "cities": [...] into their payloads (the same key fp_taxonomy and silent_activation already use), and add a line to the PR body naming these four studies as not re-run for Laurens. Whether to re-run them is a decision (below). #153 touches miss_decomposition.py in a different hunk (the registry lines), so this merges.
6. Medium-low — Phase 6 is not done, and one Phase 5 item is unstated
The body says benchmark/README.md, docs/model_comparison.md and docs/operating_point.md are not updated and gives the reason (the #55 pass). Two things go beyond that:
benchmark/README.mdis untouched by this PR and by The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153. It is the document the runbook calls "where a split actually becomes real": both split tables plus a prose section (what the split is for, its imagery, what fought the rubric, the negative-check result). After both PRs merge it still says "All nine splits are published" (line 81), "the seven US splits" (109), "All nine city splits" (117), "All eight city splits were reviewed" (502). This does not wait on the GT-completeness correction: spot-check the low-confidence incremental FPs from the operating-point curve #55 pass.BENCHMARK_SPLITSnow includeslaurens, and the HF dataset was not republished. The runbook's Phase 5 row says the allowlist is what the HF package is built from and to "Republish with build → verify → push". Not doing that is defensible — The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 renames the split, and the runbook's new naming section says to rename before the first HF publish because config names are the one irreversible piece — but it needs saying in the body as a deliberate omission, with the note that it must wait for the rename.
Stale split counts that this merge makes wrong and that #153 does not touch, for whoever does the docs pass: docs/model_comparison.md lines 3, 135, 143, 549, 766, 805, 919, 935, 1822; docs/operating_point.md lines 123, 188, 212, 373, 409, 762 (#153 rewrites the pooled block at 262–303 but not these); docs/replication.md "9 splits" at 227, 245, 282, 301, 448, 457 (#153 updates the file ledger, not the config counts).
7. Low
silent_activation.pypins its frozen tuple under the nameUS_SPLITS, the same name the live registry uses in every other module, whilefarfield_forensics.pychosePUBLISHED_SPLITSfor the same thing.from silent_activation import US_SPLITSnow returns a different tuple fromfrom miss_decomposition import US_SPLITS, and the test has to alias it (as PUBLISHED_SPLITS) to say what it means. Rename toPUBLISHED_SPLITSin the script and the test.- The
test_ap_ordering_is_not_an_artifact_of_the_floormargin, 0.0993 against an asserted 0.1, is a population mismatch as the body says — confirmed the other way round by The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153, which only moves the constants after scoring the challengers on the new split. Leaving the eight tests red rather than nudging the thresholds was the right call. - Strata 5/64/25 = 94: the geometric explanation checks out (
sample.jsonrecordsmin_spacing_m: 30,selected: 94from a 4,495-pano run), the unbiased column is 89 panos, and the body says the intervals are wider. Fine as stated.
Decisions for Jon, not fixes
- The body's "decision this needs" is moot. The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 took option 1 — all 11 challengers scored on both Laurens arms — and it is what turns this branch's CI green. The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153's own note gives the merge order: merge The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 into
benchmark/laurensfirst, confirm Add Laurens, IA — the first rural split (recall 0.390) and register it as the 8th US split #152 is green, then Add Laurens, IA — the first rural split (recall 0.390) and register it as the 8th US split #152 intomain. Merging Add Laurens, IA — the first rural split (recall 0.390) and register it as the 8th US split #152 alone puts a red commit onmain. - Reviewer confidence for
review_notes(finding 1): only the person who did the 2026-08-31 pass can supply it. Given 38 abstained miss marks and the leaf notes, "medium" would be the honest default, but that is the reviewer's call, not the reviewer's proxy's. - Re-run the four seven-split studies for Laurens, or pin them as published?
miss_decompositionandmiss_taxonomyare CPU-only from committed caches;fp_taxonomyandsilent_witnessneed challenger detections on the split, which exist only after The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 and only under the new name. Either way the answer belongs in the doc beside the numbers. - HF republish waits for the rename in The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153; say so in the body.
What holds up
- Every script-derived number in the body reproduces to the digit, and the parity gate is bit-exact as a Mapillary split should be.
- The manifest was written before review, its digest re-derives, and the three id sets agree. The archive reconciliation claim (8.3 GB Mapillary, 40 GB GSV) is the one thing a clean clone cannot check; the body says where it lives.
- The frozen-study pinning is correct and tested:
test_the_recorded_scope_is_the_pooled_populationasserts the committed payload'scitiesequals the frozen tuple, andtest_farfield_forensicsbuilds its population fromPUBLISHED_SPLITS. Registering a city can no longer restate either study. - Registration is complete on every touchpoint the runbook lists, plus
SPLIT_HEADERinscoreboard_render.py, which the runbook does not list and which The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 keeps.tier_of()resolvesGoPro Maxtoaction-modernthrough the existing"max" in modelbranch, so no new tier was needed. SERIES slot 7 is the vacated categorical slot, not an invented hue. - Camera provenance is present on all 94 records; the
copyrightfield matches the shape every other Mapillary split carries. - The eight CI failures are exactly the ones named, and refusing to bump the constants was correct.
Fix list
- [F1] High Add a
review_notesblock tobenchmark/laurens/verdicts.jsonin gainesville's shape (reviewer,reviewed_at= 2026-08-31,confidence,summary,caveats). Caveats must include: 38 of 190 missed marks abstained asunsure; leaf litter noted on three panos; strata 5/64/25 with an 89-pano unbiased column. If the reviewer's confidence is not known to you, write the block withconfidenceset to the string"unrecorded — reviewer to fill in"and say so in the PR comment; do not invent a level. Write withnewline="", keep the existing key order, and confirmscore_validation.py benchmark/laurensprints the block above the numbers — files: benchmark/laurens/verdicts.json - [F2] High Edit the PR body (
gh pr edit 152): add an amendment block under "The result" withdrawing the "misses are near, not far-field (+0.004)" bullet, citing Recall collapses on flush / minimal-reveal curb ramps: Laurens R 0.390, and the experiment that isolates it #151's 2026-09-03 amendment and The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 (laurens_gsv−0.0053); state that the y-delta, luminance and shadow figures came from an ad hoc measurement with no committed script; add a "Deliberate omissions" list naming the four studies in F5 as not re-run and the HF republish as waiting on The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153's rename; replace "The decision this needs" with a note that The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 resolves it and gives the merge order. No session links, keep the existing attribution line — files: PR body only - [F3] Medium Rewrite the stale hand prose in
docs/model_scoreboard.mdoutside the GENERATED markers so it agrees with the regenerated tables: line 3 split count; findings 1–3 (lines 7–19: the YOLO/Gemini dead-heat numbers are no longer in the headline table, RampNet's F1 range is 0.54–0.85 across eight cities and the "only stable model" sentence must go or be qualified by Laurens, RampNet's P/R are 0.951/0.686); line 24 "all ten splits"; lines 26–27 worst-city claim; line 34 caption; lines 42–44 pool description. Say "eight US splits" rather than naminglaurenswhere the sentence allows, so The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153's rename does not have to touch it — files: docs/model_scoreboard.md - [F4] Medium Regenerate
docs/figures/operating_point_pr.png(python scripts/analysis/plot_operating_point.py) anddocs/figures/storage_floor_ceiling.png(python scripts/analysis/plot_storage_floor.py), open both and check the eighth series does not collide with the legend or the curves; note in the PR comment that The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 must re-run both after its rename — files: docs/figures/operating_point_pr.png, docs/figures/storage_floor_ceiling.png - [F5] Medium Make
scripts/analysis/miss_decomposition.py,scripts/analysis/miss_taxonomy.pyandscripts/analysis/silent_witness.pywrite a"cities"list into their output payload (same keyfp_taxonomy.pyandsilent_activation.pyuse), without regenerating the committed JSONs; add a module comment in each saying the committed result covers the seven pre-Laurens splits. Add one test that every committed study JSON underanalysis_out/produced by a script with a--citiesdefault either carriescitiesor is named in an explicit allowlist with the reason — files: scripts/analysis/miss_decomposition.py, scripts/analysis/miss_taxonomy.py, scripts/analysis/silent_witness.py, tests/ - [F6] Medium-low Update
benchmark/README.md: addlaurensrows to both split tables (all-panos and unbiased, fromscore_validation.pyoutput) and a prose section in the shape of the other cities (what it is for, GoPro Max / Mapillary / 2025-11 / single uploader / 19 sequences, the leaf and abstention caveats, empty-group result), and fix the split counts at lines 81, 109, 117, 502. Name the splitlaurensas this branch does; The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 will rename — files: benchmark/README.md - [F7] Low Rename
silent_activation.py's frozen tuple fromUS_SPLITStoPUBLISHED_SPLITSand updatetests/test_silent_activation.pyto import it under that name, matchingfarfield_forensics.py— files: scripts/analysis/silent_activation.py, tests/test_silent_activation.py
🤖 Generated with Claude Code (claude-fable-5-1)
1 participant
Adds Laurens, Iowa — the first rural split — and works it through
docs/adding_a_benchmark_city.md. Companion to ProjectSidewalk/sidewalk-auto-labeler#48; findings in #149 and #151.Why this split exists (Phase 0)
Rural / small-town streetscape on a consumer 360 rig. Every existing split is urban, suburban or a college town; Laurens is 1.91 km² and 1,264 people. Pooled — US deployment city, complete verdict-grade GT.
The result
Recall is roughly half the next-worst split (clovis 0.650). It is not a weak or ramp-poor sample: Laurens is ramp-rich (2.65 ramps/pano, 3rd of 10), misses/pano 1.62 vs 0.45–1.19 elsewhere, and — uniquely among all ten splits — its misses are near, not far-field (delta +0.004 in normalized y; every other split is −0.013 to −0.030). The reviewer's leaf-litter reading did not survive measurement. Full analysis and the discriminating experiment: #151.
Parity gate: 111/111, 100.0% exact, med R 0.000 — bit-exact, as a Mapillary split should be.
Pooled numbers move
n 859 → 953 panos. Precision untouched; the recall drop is Laurens' own 0.390 entering the pool.
Two published studies pinned
farfield_forensics.pyandsilent_activation.pyread the liveUS_SPLITSbut are frozen studies — their tests pin the committed result JSONs to numbers quoted in the issues' §0c. Registering a city silently restated both published findings. Each now carries its own explicit split tuple with the reason. This is a latent trap for every future split, not something specific to Laurens, and is probably worth a line in the runbook.Checklist
Phase 1 — bundle
d8dd392b…)export_benchmark.py --bundleexited zero;index.csvwritten, zero decayrecords.jsonlcommitted;panos/archived durably on makelab2 (8.3 GB Mapillary + 40 GB GSV, both reconciled 1:1, zero decay)_spreadwalks the whole pool and the town saturates at 64 detection panos held 30 m apart. Unbiased column is 89 vs the usual 120, so intervals are wider. Strata definitions and de-clustering unchanged.Phase 2–3 — ground truth
gt_gallery.pyat model resolutionreview_noteswritten · [x]verdicts.jsoncommitted · [x]score_validation.pyrun, unbiased recordedGoPro/GoPro Max, all 94)imagery_manifest.jsonwritten before review (7485886ea08f671b) — a step the runbook still does not mentionPhase 4 — operating point
sweep/hist/gtbias/floor/distancere-run;op_cache/laurens.json+op/*.csvcommittedcorrected/tagcheck— blocked on the abovePhase 5 — code
US_SPLITS/CITY_SPLITS/ALL_SPLITS· [x]TIER· [x]BENCHMARK_SPLITS· [x]SPLIT_HEADERHELD_OUT— n/a, pooledtier_of()already resolves GoPro Max →action-modernSERIESslot 7 (#4a3aa7), vacated by the manual_gold fold — no invented huepytest -q— 8 failures intest_scoreboard.py, deliberately not silenced (see below)Phase 6 — docs
benchmark/README.md,docs/model_comparison.md,docs/operating_point.md— not updated; the corrected tables need the GT-completeness correction: spot-check the low-confidence incremental FPs from the operating-point curve #55 pass, and the scoreboard question below changes what the model-comparison text should say.The decision this needs
No challenger model has been scored on Laurens. Consequences, all real rather than cosmetic:
test_ap_ordering_is_not_an_artifact_of_the_floornow compares RampNet's 8-split AP (0.829) against YOLO's 7-split AP (0.730) — margin 0.0993 against an asserted 0.1. That is a population mismatch, not a stale constant, and bumping the threshold would paper over exactly the artifact the test exists to catch.Two ways out — your call, which is why the constants are untouched:
🤖 Generated with Claude Code (claude-opus-5[1m], effort: high)