Make the seed settable, pre-register its reading, and launch both seed-variance campaigns (#51, #135)

[jonfroehlich]

Makes the training seed settable in both trainers, pre-registers how its variance will be read, and launches the two campaigns that measure it. Both campaigns are running as of 2026-09-03 — job IDs below.

Why now

#51 and #135 hit the same wall from opposite sides.

• Train a supervised YOLO baseline (YOLO11 / YOLO26) on the RampNet dataset — isolate architecture vs. data #51: the matched-operating-point read (The RampNet-vs-YOLO gap at matched operating points is 0.039, not 0.252 — plus the label rescue and the equirect control (#51) #154) put the RampNet-vs-YOLO residual at 0.039 F1, not 0.252. Every arm in that comparison is seed: 0 — the Ultralytics default — in all eight args.yaml files. Train a supervised YOLO baseline (YOLO11 / YOLO26) on the RampNet dataset — isolate architecture vs. data #51's own rule is that differences under ~0.02 should not be read.
• Run B (#84): the gate passed — decide whether to spend 1,675-3,350 GPU-hours on the annealed arm #135: the power analysis measured a paired MDE of 0.0063 and then said plainly that the binding limit is unmeasured seed variance, n=1.

Neither trainer could vary the seed at all: train.py hardcoded manual_seed(42) with no flag, and run_yolo_train_tillicum.slurm never passed one to Ultralytics.

Everything downstream inherits this. With no noise floor, no RampNet 2.0 improvement can be called real either.

The half that was easy to get wrong

Stage 2 has two sources of run-to-run randomness and only one was ever set. DistributedSampler carries its own seed (default 0) and derives each epoch's permutation from seed + epoch inside set_epoch(), independent of torch.manual_seed. A sweep that moved only the torch seed would reuse one data order across every replicate — understating the spread, and doing it silently, since no log line distinguishes the two cases.

So both seeds move together, with one exception that has to be exact: at the historical seed 42 the sampler must stay at its historical 0, or the default stops reproducing published runs. That pairing is rampnet/seeding.py::sampler_seed_for, tested rather than commented.

Why the klone launcher is a new file

train.py writes best_model.pth and latest_checkpoint.pth to the current directory, not to --checkpoint-dir. Three replicates launched from one directory would overwrite each other's best model and — worse — each other's resume state: a resume file from another seed loads silently as if it were this run's own, converging the arms onto one lineage with nothing in the log to say so. run_train_seed.slurm gives each replicate its own RUNDIR and cds into it. run_train.slurm stays untouched as the preserved record of the published run and #135's rungs, the same reasoning that kept the Tillicum YOLO launcher separate from its klone original.

Pre-registered, before any replicate finished

docs/seed_variance_51_135.md fixes the decision rule on the sample SD s of the three Campaign A replicates:

s	reading
≤ 0.010	0.039 is ≈4σ — the architecture advantage is small but real
≥ 0.020	0.039 is inside 2σ — #51 closes with "the supervised baseline is statistically indistinguishable from RampNet at matched operating points"
0.010–0.020	ambiguous; report the interval, claim neither, combine with Campaign B

The second branch makes our own headline smaller and is accepted in advance.

What is running

Campaign A — YOLO seed variance, Tillicum. Three replicates of y11x_tiles at seeds 1/2/3, config identical to the as-run args.yaml in every respect except the seed. Jobs 274367 (running, verified seed=1 in its log header), 274369, 274370, each with CHAIN=5 to cover the 24 h normal-QoS ceiling. ~3.0 h/epoch on one H200; ~$21.60 per 24 h link, ≤$389 for all three, against a $1,500/month cap with $0 used this cycle.

Tillicum was chosen over free klone for a measured reason: the tiles arm consumes 8.5 MB/s there against a filesystem measured at 8.3–11.8 MB/s — it sits on the ceiling — so three concurrent replicates would contend for the wall itself. On Tillicum the same arm uses 4% of available bandwidth and is genuinely GPU-bound.

Campaign B — RampNet seed variance, klone. Three replicates of the committed recipe (1 epoch / 9,378 steps, constant lr 1e-5, global batch 16) at seeds 1/2/3. Jobs 39515025/26/27 on ckpt-all, free and preemptable, resumed by --requeue plus train.py's own latest_checkpoint.pth. Calendar, not money, is the risk: ckpt-all's duty cycle was 3.9% in 2026-08.

One deliberate deviation: save_period=1

The #51 arms ran save_period: -1, which is exactly why the epoch-curve follow-up had to be retracted — no per-epoch weights exist for any arm and they cannot be recovered. Keeping every epoch costs ~20 GB against a 1 TB allocation and buys back an analysis that is currently foreclosed.

Tests

tests/test_seeding.py (21 tests) checks the plumbing at the source level, because every failure mode here is silent:

• the default still maps to sampler seed 0, so reproductions are unchanged
• all three RNGs follow the flag, and no 42 literal survives
• the launcher refuses to default SEED — an unset replicate would be a silent duplicate of the published run
• per-seed RUNDIR isolation, and no checkpoints under klone's 10 GB home quota
• the Tillicum heredoc's positional list still lines up with its sys.argv unpack — an off-by-one there shifts imgsz into epochs and trains a wrong model that finishes green

Full suite: 1,343 passed, 1 skipped. --seed verified end to end in the real klone sidewalkcv2 environment, and seed=1 confirmed in job 274367's Ultralytics arg dump.

Stated limitations

• n=3 gives a wide interval on the SD itself (95% CI on σ spans ~0.5σ̂–3.7σ̂). A 4th and 5th replicate are ~$119 each and are the pre-registered response if the result lands in the ambiguous band.
• Campaign A's replicates are Tillicum H200; the existing seed: 0 arm is klone L40S. The SD is over the three same-hardware replicates only; the seed-0 arm is a separate cross-hardware check, not a fourth sample.
• Only y11x_tiles is replicated — the measured SD is not automatically the other arms'.
• Seed variance is not the same as training-run variance; Campaign B has no deterministic=True guarantee, so its spread is an upper bound on seed effect alone.

Related: #154 (the parity result this exists to adjudicate), #51, #135.

🤖 Generated with Claude Code (claude-opus-5[1m])