The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151)

[jonfroehlich]

CI note, and it settles the merge order. #152 is red on its own (pytest fails on 3.10 and 3.12): it adds laurens to ALL_SPLITS while model_comparison.md still lacks the challenger rows, and the pooled AP/precision constants in tests/test_scoreboard.py had not been updated for the new split. This PR is what makes it green — it adds the 22 detection files, the doc rows and the updated constants. So merge this into benchmark/laurens first, confirm #152 goes green, then merge #152 into main. Merging #152 alone would put a red commit on main.

Stacked on #152 — base is benchmark/laurens, not main. Merge #152 first, then this. (We have been bitten before by a stacked PR being merged into its base branch and never reaching main; check main afterwards.)

#152 adds Laurens as the 8th US split and records its recall of 0.390, the benchmark's worst by a factor of two. This PR adds the second imagery arm over the same footprint — the experiment #151 pre-registered to discriminate rural ramp geometry from rig and season — reviews its ground truth, and scores all 11 challengers on both arms.

What it answers

#151 pre-registered a decision rule: GSV recall ≈ 0.35 → ramp geometry, ≈ 0.75 → rig or season. The GSV arm reads P 0.933 / R 0.509 / F1 0.659 (86/86 panos reviewed, 220 GT ramps; verdict scorer 0.505, Wilson 0.439–0.570). That is neither, so the answer is the split verdict the issue listed as a real possible outcome — and it decomposes into two halves that point in different directions.

Rig/season is real, worth ~12 recall points. Confirmed on the corners both rigs actually saw, so it is not a sampling artifact:

radius	panos (mly/gsv)	mly R	gsv R	Δ
15 m	33 / 33	0.415	0.545	+0.130
20 m	49 / 51	0.393	0.518	+0.125
25 m	54 / 57	0.398	0.502	+0.104

The deficit mostly survives anyway. 0.509 is still the worst US split by ~18 points (next lowest paterson 0.684). Leaf-free September GSV at 16384×8192 does not rescue Laurens, so roughly a third of the gap is the camera and two thirds is the place.

The specific evidence for flush ramps weakened. The inverted near-miss delta — the thing that made Laurens unique among ten splits — is arm-specific: laurens_mapillary +0.0038, laurens_gsv −0.0053, back to the normal far-field direction. #151's body is amended and that bullet withdrawn.

The rig, not the town

Every model was run through the identical leg list on both arms, so imagery is the only thing that moves. Every zero-shot challenger is flat or worse on the arm RampNet prefers — seven of eight move down, none moves up by more than 0.008 — while RampNet gains +0.115 (F1 0.543 → 0.659). If rural streetscape were intrinsically hard, the arm that is easier for RampNet would be easier for them too. Laurens' 0.390 is RampNet meeting an out-of-domain rig, not a town that defeats detectors.

Two consequences worth flagging to reviewers:

• The RampNet–YOLO ordering flips between the arms. On Mapillary the supervised baseline wins (y26_pano 0.574, y11l_pano 0.563 vs 0.543); on GSV RampNet leads by 0.072 (0.659 vs 0.587). "YOLO beats RampNet on the rural split" is really "on non-GSV imagery". This qualifies Train a supervised YOLO baseline (YOLO11 / YOLO26) on the RampNet dataset — isolate architecture vs. data #51 rather than contradicting it.
• Being GSV-trained is not the whole mechanism. The YOLO arms trained on the same GSV-derived dataset barely move (+0.039, +0.024, and y26_pano goes the other way) against RampNet's +0.115, so RampNet is markedly more rig-sensitive than a YOLO trained on its own data. That points at preprocessing and the 2048×4096 input rather than the training distribution — an open question this split hands to Recall collapses on flush / minimal-reveal curb ramps: Laurens R 0.390, and the experiment that isolates it #151 and Experiment: curb-ramp detection performance vs. input resolution (Mapillary high-res) #25, not one it closes.

Registration

laurens_gsv is held out of the pooled basis for non-independence, not GT quality: 59% of its panos sit within 20 m of a laurens_mapillary one (median NN 17.2 m), so the arms largely see the same physical ramps and pooling both would double-count them. The pooled operating point is unchanged, which is what held-out has to mean. laurens is renamed laurens_mapillary since it is now one of two arms, and CITY_OF plus the one-pooled-split-per-city test — both prescribed by docs/adding_a_benchmark_city.md and neither of which existed — are now real.

Three latent bugs, each found by a test rather than by reading

• pool_of() named the three held-out splits literally instead of deriving from HELD_OUT, so a newly held-out split was pooled anyway — laurens_gsv would have been pooled alongside its own sibling arm while HELD_OUT said otherwise. Now derived.
• The AP provenance table printed a false provenance claim. Its binary (substituted, or "bundle, already at 0.05") held only while manual_gold was the sole unsubstituted row; it read "already at 0.05" against laurens_gsv, whose bundle is truncated at 0.55 with no op_cache to swap in. Cells now carry bundle_floor and the table distinguishes three cases.
• docs/replication.md's file ledger went stale again (125 → 136).

Caveats, carried in the docs beside the numbers

• The cross-arm ΔF1 table is unpaired — 249 GT ramps against 220. The ~51 corners inside 20 m are the paired subset that would make it a measurement, and both arms' detections are committed, so that needs no new inference. The recall half is already done (the radius table above).
• AP is not comparable across the arms for RampNet: laurens_mapillary has a 0.05-floor op_cache and laurens_gsv does not, so 0.494 is truncated at 0.55 while the other arm's 0.691 is not. P/R/F1 are comparable — both read at 0.55.

Detections: 22 new files, all --verify identical to the cache.

Closes nothing on its own — #151 stays open on the flush-ramp escalation and the rig-sensitivity question it opened.

🤖 Generated with Claude Code (claude-opus-5[1m])

[jonfroehlich]

Deep review

Re-ran everything from the branch head (d388f17) in a clean detached worktree on Windows, against the committed inputs only. The result holds and the numbers are right: the scoreboard regenerates byte-for-byte (scoreboard.json, model_scoreboard.md, all five PNGs — git status empty afterwards, --check current); the full suite is 1,325 passed, 1 skipped, including the two population tests that were red on #152; every cell in the two split tables and the twelve-row ΔF1 table matches analysis_out/scoreboard.json; and the radius table, the 59% and the 17.2 m — which no committed script produces — reproduce exactly when re-derived with rampnet.validation.collect over the pano subsets. The branch head did not move during the review.

What follows is mostly provenance and one real bug, not the science. Merge against main is clean; it conflicts with #146 on the regenerated files only (details under decisions).

---

1. Medium — plot_operating_point.py no longer runs, and the figure it owns is two splits stale

laurens_gsv was added to ALL_SPLITS (low_floor_sweep.py:70) with no analysis_out/op_cache/laurens_gsv.json. low_floor_sweep.py tolerates that — it guards every cache read with os.path.exists (lines 708, 896, 1083), which is why the six op/*.csv regenerated with zero laurens_gsv rows. plot_operating_point.collect() (line 87) does not:

File "scripts/analysis/low_floor_sweep.py", line 386, in load_split
    panos, meta = read_cache(os.path.join(cache_dir, f"{city}.json"))
FileNotFoundError: ... analysis_out/op_cache/laurens_gsv.json

So the SERIES / HELD_DASH / LABEL entries this PR adds for laurens_gsv (plot_operating_point.py:70,79,81) are unreachable, and docs/figures/operating_point_pr.png — embedded at operating_point.md:248, directly above the table this PR re-pooled to eight splits — was last written at bdd7d55 (the sao_paulo registration). It still shows the seven-split pool with no Laurens series of either kind. Also: the § mark chosen for the laurens_gsv legend label is the mark operating_point.md:308–313 already uses for sao_paulo.

Fix: skip splits with no cache in collect() and say so on stdout; regenerate the figure from the committed op_cache (CPU, matplotlib); pick a legend mark that is not already taken in the doc.

2. Medium — three unexplained artifacts from a different issue rode in on the rename commit

analysis_out/crop_window_eval_{annapolis,paterson,sao_paulo}.json (1,181–1,261 lines each) were added by 8d61982 "Rename laurens -> laurens_mapillary". The body does not mention them; no doc, test or script references them (git grep crop_window_eval_paterson is empty; docs/crop_window_eval.md covers richmond only); they carry "issue": 114, not #151; and each declares its input as benchmark/<city>/boxes.json, which is not in the repository (ls fails for all three). They cannot be regenerated from a clean clone and nothing explains them.

Fix: git rm the three files from this branch. They belong in a #114 PR that also commits the boxes.json inputs and a line in docs/crop_window_eval.md.

3. Medium — the six paid legs are not in the committed ledger, and no leg has a time

analysis_out/usage_log.jsonl on this branch contains zero rows for either Laurens arm. The seven rows that were written (six Gemini legs plus a 2-pano smoke on gemini-3.6-flash) exist only as uncommitted lines in the main checkout's working tree:

ts (UTC, 2026-09-01)	bundle	leg	panos	calls	est. cost
12:32	laurens_mapillary	gemini-3.6-flash (smoke)	2	12	$0.03
13:14	laurens_mapillary	gemini-3.1-pro-preview	94	564	$1.54
13:15	laurens_mapillary	gemini-3.6-flash	94	402	$0.99
13:35	laurens_mapillary	gemini-3.7-flash	94	564	$1.10
14:05	laurens_gsv	gemini-3.1-pro-preview	86	516	$1.38
14:16	laurens_gsv	gemini-3.6-flash	86	516	$1.23
14:26	laurens_gsv	gemini-3.7-flash	86	516	$0.96
		total			$7.22

This is the #143 failure mode again: the rows were written, to a file that is not on the branch that publishes the detections. None of the 22 legs has a wall-clock figure (the rows predate #147's timing fields; the sixteen GPU legs — A40 and 2×L40S per replication.md:139–146 — wrote no row at all), and docs/replication.md's Laurens section states no cost.

Fix: commit the seven rows into analysis_out/usage_log.jsonl on this branch (they can be read from the main checkout's working tree with git diff -- analysis_out/usage_log.jsonl without modifying anything; de-duplicate on ts if they have since landed elsewhere), and add one line to the Laurens section of replication.md: $7.22 across six Gemini legs, wall-clock unrecorded for all 22 legs, GPU-hours for the sixteen local legs unrecorded — as a stated gap, not silence.

4. Medium — the radius table and the overlap statistics have no script

The numbers are right. Recomputed from records.jsonl coordinates (haversine) and verdicts.json through rampnet.validation.collect on each subset:

radius	panos mly/gsv	mly R	gsv R	Δ
15 m	33/33	0.415 (59/142)	0.545 (79/145)	+0.129
20 m	49/51	0.393 (79/201)	0.518 (99/191)	+0.125
25 m	54/57	0.398 (86/216)	0.502 (103/205)	+0.104

gsv→mly nearest neighbour: median 17.2 m, 51/86 = 59.3% within 20 m. Every cell matches the body (the 15 m delta is +0.129 unrounded; the body's +0.130 is a subtraction of rounded cells). But that is my 40-line script in a scratch directory, not one in the repo — the table appears in the PR body, test_miss_decomposition.py:131, low_floor_sweep.py:84, adding_a_benchmark_city.md and model_comparison.md, and the only thing standing behind it is prose.

The "rig, not the town" ΔF1 table is likewise hand-assembled from scoreboard.json. Eleven of twelve deltas match the unrounded values; y11l_pano reads +0.024 where the unrounded delta is +0.023 (0.5870 − 0.5634), because the table subtracts rounded cells.

Fix: a committed scripts/analysis/laurens_arm_overlap.py that writes analysis_out/laurens_arm_overlap.json (nearest-neighbour distances both directions, the three radius rows, the twelve ΔF1 rows from scoreboard.json), a test pinning the counts and recalls above, and the doc table regenerated from it.

5. Medium — the doc argues both sides sixty lines apart

Both passages are new in this PR (model_comparison.md). The laurens_mapillary paragraph, point 1: "the rural deficit is not RampNet-specific in the zero-shot field — whatever makes this town hard makes it hard for everything that has never seen a curb ramp label." Then "The rig, not the town", point 1: "The town is not the problem." The second is the finding; the first is the pre-GSV-arm reading it replaced, still stated as a conclusion.

Fix: amend the first to say the GSV arm reversed it and point to the section below, or delete the sentence.

6. Medium — eight collapsed line continuations in the new runbook block

docs/replication.md lines 139–143, 146, 150 and 151 (all from 5111834) are single physical lines with five-space gaps where \-newline used to be, e.g.

python scripts/model_comparison/compare.py benchmark/laurens_mapillary     --models rampnet,owlv2,gdino --sweep

This is exactly the class #136 guards against, and #136's test walks replication.md. I ran that test against this branch: it fails (first on the pre-existing model_comparison.md:1306, which #136 itself repairs, then on these eight). Once #136 and this PR are both on main, the suite is red.

Fix: restore the continuations, written via chr(92) so no patch layer eats them (as #136 documents).

7. Low — the GSV headline P/R and the CI beside it come from two scorers

scripts/score_validation.py benchmark/laurens_gsv (the runbook's Phase 3 instrument) reports P 0.925 / R 0.505 (CI 0.439–0.570): 111 correct, 9 incorrect, and one detection marked duplicate, scored as a false positive by default. The doc table and body say 0.933 / 0.509 (112/8/108) — the scoreboard's scorer, which counts that one duplicate as a hit. The body then quotes the 0.505 CI next to the 0.933 precision. The mapillary arm has no duplicate mark, so its two scorers agree and nobody would notice the split.

Fix: one sentence under the laurens_gsv table naming the scorer and the one-detection difference.

8. Low — replication gaps not stated where the numbers are

• Neither arm's panoramas are published (Add Laurens, IA — the first rural split (recall 0.390) and register it as the 8th US split #152: 8.3 GB Mapillary and 40 GB GSV on makelab2). replication.md's Laurens section does not say so or name what unblocks it — which per this PR's own runbook text is an HF publish under the new config names, before the first publish.
• benchmark/README.md:81 still says "All nine splits are published" and :109 "the seven US splits"; neither arm is listed.
• The split coverage matrix at the top of model_comparison.md (the | annapolis | ✅ ... table) has no row for either arm.
• operating_point.md:212, 419, 772 still say "seven US splits" for the tag-based sections. That is correct — the GT-completeness correction: spot-check the low-confidence incremental FPs from the operating-point curve #55 pass has not run on Laurens — but it now sits under a table that is eight-split, with nothing saying the two populations differ.

Fix: the four statements, each where the reader is.

9. Low — rename leftovers inside the moved bundle

benchmark/laurens_mapillary/verdicts.json still carries run_key: "laurens", run_name: "laurens", source: "benchmark\laurens\records.jsonl"; imagery_manifest.json has "city": "laurens". box_gallery.py:1047 reads run_key. The manifest digest is over pano hashes only (imagery_manifest.py:48, digest_of), so correcting city leaves 7485886ea08f671b unchanged.

Fix: update the four fields.

10. Low — register

replication.md:155 (new): "load-bearing". Out of scope but in the same file: :588 "naming wart", and adding_a_benchmark_city.md:224 "load-bearing".

---

Decisions for Jon, not fixes

• A 0.05-floor op_cache for laurens_gsv. It needs the GSV panos (makelab2) and a model pass, not just scoring. With it the AP column becomes comparable across the arms, the operating-point figure can carry the arm, and the "AP is not comparable" caveat can go.
• The paired ~51-corner analysis. Finding 4's script gives the recall half for free (it already does: +0.125 at 20 m). Whether the precision/F1 half lands here or in Recall collapses on flush / minimal-reveal curb ramps: Laurens R 0.390, and the experiment that isolates it #151 is a scope call.
• Merge order and the one conflict. Clean against main, Record time as well as money for every non-free run (#143) #147, Vistas at resolution parity: the handicap was real, and it was recall (#126) #137, Actually fix the two collapsed runbook snippets from #129, and guard the class #136 and The RampNet-vs-YOLO gap at matched operating points is 0.039, not 0.252 — plus the label rescue and the equirect control (#51) #154. Against claude-opus-5 on nine splits: the annapolis displacement does not generalise (#139) #146 it conflicts on analysis_out/scoreboard.json, docs/model_scoreboard.md, docs/replication.md and four scoreboard_*.png — all regenerated files. Whichever lands second re-runs scoreboard.py and takes the fresh output; nothing to resolve by hand except the replication ledger paragraph.
• The body's "next lowest paterson 0.684" matches neither committed source: scoreboard.json has 0.686 (bundle detections), operating_point.md:304 has 0.681 (op_cache at 0.55).
• Reviewer attestation is prose only. Neither arm's verdicts.json has the review_notes block score_validation.py prints when present (reviewer, date, confidence, caveats), so "reviewer confidence HIGH" exists only in the doc. Every other split is the same, so this is benchmark-wide rather than this PR's defect — and only the reviewer can write it, which is why it is not on the fix list.

What holds up

• The scoreboard is exactly what the script produces, on Windows: JSON, rendered doc and all five figures byte-identical; --check current.
• Every number in both split tables and the twelve-row ΔF1 table matches scoreboard.json, recomputed independently: RampNet +0.115 (0.5434 → 0.6588); the eight zero-shot challengers move −0.064, −0.049, −0.048, −0.032, −0.020, −0.007, −0.003 and +0.008 — seven down, none up by more than 0.008, as written; the ordering flip (y26 0.574 / y11l 0.563 vs 0.543 on Mapillary; 0.659 vs 0.587 on GSV); the YOLO deltas +0.039 / +0.024 / −0.036; AP 0.691 (op_cache) vs 0.494 (bundle, floor 0.552).
• The radius table and the overlap numbers reproduce exactly with the repo's own verdict scorer (finding 4 is about the missing script, not the numbers).
• pool_of() is now derived from HELD_OUT and "found by a test" is true: test_pool_holds_out_budapest_and_gold_by_default asserts set(pool) == set(US_SPLITS), which the old literal list would have failed the moment laurens_gsv entered CITY_SPLITS.
• The three-way AP provenance (bundle_floor, scoreboard.py:356; ap_provenance_table, scoreboard_render.py:213–224) and its test check the real cells: manual_gold floor < 0.4, laurens_gsv floor 0.552 with ap == ap_bundle.
• CITY_OF and both guards in test_miss_decomposition.py do what the body says; the prefix heuristic in the second one resolves every current split correctly.
• The 22 detection files: identical top-level schema to the existing files, signature["model_id"] equal to the published label on all 22, eleven legs × two arms with nothing missing. The --verify claim cannot be checked here — no .model_cache — and is taken on the body's word.
• The replication ledger arithmetic: 82 + 12 + 36 + 4 + 2 = 136 files on disk.
• Bundle order: imagery_manifest.json committed (62cc220) before verdicts.json (e9dfd86); tier_of() resolves the GSV arm via source: "launch"; no ToS or licensing text anywhere in the diff.
• CI: the body's "this is what makes Add Laurens, IA — the first rural split (recall 0.390) and register it as the 8th US split #152 green" is true — test_ap_ordering_is_not_an_artifact_of_the_floor and the threshold-mark test both pass with the challengers on eight splits.

Fix list

• [F1] Medium plot_operating_point.collect() skips splits with no op_cache file and prints which; regenerate docs/figures/operating_point_pr.png from the committed cache; give laurens_gsv a legend mark not already used by operating_point.md (§ is sao_paulo's) — files: scripts/analysis/plot_operating_point.py, docs/figures/operating_point_pr.png
• [F2] Medium git rm the three unexplained Score crop-window rules against the manual_gold boxes: containment, margin, and context ratio (crop-clarity acceptance test) #114 artifacts — files: analysis_out/crop_window_eval_annapolis.json, analysis_out/crop_window_eval_paterson.json, analysis_out/crop_window_eval_sao_paulo.json
• [F3] Medium commit the seven Laurens ledger rows (read them from the main checkout's working tree with git diff, never modify that checkout; de-duplicate on ts) and add the cost/time line to the Laurens section: $7.22 over six Gemini legs, no wall-clock for any of the 22 legs, no GPU-hours for the sixteen local legs — files: analysis_out/usage_log.jsonl, docs/replication.md
• [F4] Medium add scripts/analysis/laurens_arm_overlap.py (nearest-neighbour distances both ways, the 15/20/25 m radius-recall rows via rampnet.validation.collect, the twelve ΔF1 rows from scoreboard.json) writing analysis_out/laurens_arm_overlap.json (LF, rounded floats, whitelisted in .gitignore), a test pinning 33/33 · 49/51 · 54/57, 0.415/0.545, 0.393/0.518, 0.398/0.502, 59.3%, 17.2 m; regenerate the doc's ΔF1 table from it (y11l_pano becomes +0.023) — files: scripts/analysis/laurens_arm_overlap.py, analysis_out/laurens_arm_overlap.json, tests/test_laurens_arm_overlap.py, docs/model_comparison.md, .gitignore
• [F5] Medium reconcile laurens_mapillary point 1 ("whatever makes this town hard makes it hard for everything") with "The rig, not the town" — amend to say the GSV arm reversed the reading and point down — files: docs/model_comparison.md
• [F6] Medium restore the \ continuations on replication.md lines 139–143, 146, 150, 151 (write via chr(92)); confirm with Actually fix the two collapsed runbook snippets from #129, and guard the class #136's test_no_runbook_snippet_has_a_collapsed_line_continuation logic — files: docs/replication.md
• [F7] Low one sentence under the laurens_gsv table: table is the scoreboard scorer (112/8/108, P 0.933 / R 0.509); score_validation.py reads 0.925 / 0.505 because its one duplicate mark scores as a false positive; the CI quoted is the latter's — files: docs/model_comparison.md
• [F8] Low state the replication gaps where the numbers are: Laurens panoramas unpublished for both arms (makelab2; unblocked by an HF publish under the new config names) in replication.md's Laurens section; benchmark/README.md nine→twelve bundles with both arms listed and marked unpublished, "seven US splits"→eight; a row for each arm in model_comparison.md's coverage matrix; a one-line note at operating_point.md's tag-based "seven US splits" sections that they predate Laurens — files: docs/replication.md, benchmark/README.md, docs/model_comparison.md, docs/operating_point.md
• [F9] Low update run_key/run_name/source in benchmark/laurens_mapillary/verdicts.json and city in benchmark/laurens_mapillary/imagery_manifest.json to laurens_mapillary (digest unchanged; verify with imagery_manifest.py's digest_of) — files: benchmark/laurens_mapillary/verdicts.json, benchmark/laurens_mapillary/imagery_manifest.json
• [F10] Low replace "load-bearing" at replication.md:155 with plain wording — files: docs/replication.md

🤖 Generated with Claude Code (claude-fable-5-1)