fix(data): every normalization reduces over a column's measured rows (#726)

[wshlavacek]

Closes #726. Completes the sweep #479 started.

The rule, stated once

A sparse column now normalizes to exactly what the dense column of its measured values would. Every reduction (max, min, mean, std) and every reference row (argmax, argmin, the init/unit baseline) skips the NaNs, so a NaN row stays a NaN row and moves nothing else. #479 established that for peak and floor; zero, init and unit were missed, and neither issue covered a column with no measured value.

That invariant is the main test: a property over random draws comparing a sparse column against the dense column of its measured values, per method.

Defect A — a NaN at an unscored row discarded a good parameter set

A simulated column can legitimately carry NaN at some output rows — one failed integration step, or an observable that is 0/0 at t=0 — while the exp file measures a handful of the simulated times, so that row is frequently one nothing scores.

normalize_to_zero reduces the whole column, so one NaN made the mean NaN → every centered value NaN → the std NaN. _subtract_baseline read row 0, so a NaN there did the same to init and unit. The column then scored as a failed simulation and the parameter set was thrown away, with nothing logged to say a normalizer rather than the model made the call.

peak on the identical input is the control that makes this a defect rather than a policy:

unnormalized -> 0.04                  (scores fine)
peak         -> 38.667                (NaN confined to its row -- nan-aware since #479)
zero         -> None                  <- whole column NaN -> FAILED SIMULATION
init         -> None                  <- same
unit         -> ValueError: All-NaN slice encountered

init / unit now take the first measured row as their baseline, which is what "the initial value" meant. _subtract_baseline returns the row it used so normalize_to_unit_scale records the real baseline_row — the gradient reads that baseline back by index, so recording 0 would have been a wrong row.

Defect B — an all-NaN column raised instead of being scored as a failure

np.nanargmax / np.nanargmin raise ValueError: All-NaN slice encountered rather than returning a sentinel, so peak, floor and unit crashed outright and zero / init reached the same state by poisoning. For this case #479's nan-awareness had converted silent corruption into an unexplained traceback.

Such a column is now left untouched with an identity record. Normalization is a transform; whether an all-NaN simulated column is a failure is scoring's call, and scoring already makes it. Both fit scoring paths absorbed the exception in #388's blanket handler and penalized the evaluation with inf, so no fit ever died — but two callers lack that handler:

• model_check.run_check calls normalize outside the try below it (which covers only postprocess_data), and neither it nor its caller in pybnf.py catches anything. --check-simulation on a model producing an all-NaN observable died with a numpy traceback, bypassing the 'Simulation contained NaN or Inf values' diagnostic three lines later that the command exists to print.
• the config-load floor on experimental data — an exp file carrying a wholly unmeasured observable column (the shape ave_norm_sos's column average is NaN-poisoned by any missing data point, turning the whole objective into NaN #707 supports) crashed before the fit began.

Neither caller grows a handler; the transform stops raising instead.

The gradient moves with it

gradient/assembly.py's z-score rule reduced over every row and divided by nrows - ddof. Once a partially-NaN column scores it also reaches the gradient, where those full-column reductions return NaN and poison the whole Jacobian. They now mask to the measured rows — read off normed, which marks them — and count the same points the value-side std did. That keeps record.ddof at its plain user-facing meaning instead of smuggling the NaN count into it.

peak / init / unit need no gradient change: their rules read two or three specific rows rather than reducing the column, and nanargmax never returns a NaN row.

Tests

file	what
test_data_class.py	the sparse-equals-dense invariant as a property over random draws, per method; a hand-computed sparse case pinning the recorded scale / ref_row / baseline_row; the untouched all-NaN column
test_gradient_assembly.py	the existing finite-difference oracle extended to a NaN at an unscored row, on the baseline row and mid-column
test_model_check.py	an all-NaN column reaches the NaN message rather than a traceback — driving the real Result.normalize, not the file's recording fake

Against the old code: 3 of the 5 property cases fail (peak and floor pass — the controls), 4 of the 8 finite-difference cases, and all 3 model-check cases.

Full suite with BNGPATH set: 5111 passed, 25 skipped. Docs rebuilt clean under -W --keep-going.

Behaviour and scope

On a dense column every reduction is the plain one it always was, so existing fits are byte-identical.

Two degenerate cases are unchanged and left as they were: a z-score over a single measured value has an undefined std, and unit on a constant column divides by 0. I verified both against the old code — they behave exactly as the equivalent dense column always has, so this change makes them reachable from sparse data rather than introducing them. Fixing either would change dense behaviour and belongs in its own issue.

A correction to the original framing

The report this came from described a sparse multi-observable exp file reaching normalize_to_zero. It cannot: Configuration._SYMMETRIC_NORMALIZATIONS is frozenset({'floor'}), and only the floor is applied to experimental data — peak / init / zero / unit are simulation-side only. The exp side is genuinely implicated, but through floor alone (defect B, second bullet). #726 is written with the corrected framing.