Skip to content

fix(objective): ave_norm_sos's normalizer averages the measured points only (#707) - #725

Merged
wshlavacek merged 1 commit into
mainfrom
fix-707-ave-norm-sos-nanmean
Sep 17, 2026
Merged

wshlavacek merged 1 commit into
mainfrom
fix-707-ave-norm-sos-nanmean

Conversation

@wshlavacek

Copy link
Copy Markdown
Collaborator

Closes #707.

The defect

AveNormSumOfSquaresObjective.evaluate precomputed each column's normalizer with
np.average over the raw column, NaNs included. A NaN in an experimental column is
missing data and a first-class supported value — the reader parses it natively,
docs/config.rst tells users to write it, SummationObjective.evaluate skips those rows,
and #479 already made every Data normalization reduce past them, because a sparse
multi-observable column carries NaN wherever that observable was not measured.

But this mean is a divisor. Over the raw column it was NaN, so ((sim - exp) / ybar)**2
returned NaN for every present point of the column too, and the whole objective went NaN.

Nothing downstream recovered it. evaluate gates the prediction for NaN/inf but not the
accumulated func_value, so it returned a float rather than None — a NaN is not a failed
simulation, so it never took the failed-simulation path and reached the optimizer as a real
score. Every NaN comparison is False, so new_score < best_score was never true: every
parameter set was silently rejected, and the fit ran to completion reporting no improvement,
with no error and nothing in the log to say why.

Scope: four sites, not two

The issue named two. Four computed this same column mean, and all four had it:

site consequence
pybnf/objective.py — legacy objfunc = ave_norm_sos whole objective NaN
pybnf/noise/source.pyColumnMeanSigma, i.e. modern objective = ave_norm_sos same
pybnf/petab/export.py writes the literal nan as an observable's noiseFormula — not a usable PEtab problem
pybnf/petab/import_.py compares a constant sigma against the column mean to tell ave_norm_sos from sos; NaN compares equal to nothing, so a round-tripped ave_norm_sos came back silently downgraded to sos

The two PEtab sites are not hypothetical: PEtab's own round-trip writes nan for an
unmeasured point (import_.py:1219), so a sparse column is exactly what arrives there.

ColumnMeanSigma.value's comment claimed its result was identical to the legacy path. It
was — poisoning included. All four now read one pybnf.data.observed_mean, so that claim is
structural rather than two parallel implementations that happened to agree.

Behaviour

On a dense column observed_mean is the plain mean, so every existing fit and golden value
is byte-identical
. A column with no measured point has no mean and gives NaN; harmless,
since every one of its rows is skipped in scoring and the normalizer is never read. It is
written out rather than using np.nanmean so that case returns quietly instead of warning
Mean of empty slice once per unscored column on every evaluate() of the fit.

Tests

The NaN-in-experiment case was untested for this objfunc — the one existing ave_norm_sos
NaN test puts the NaN in the simulation. Added:

All four behavioural tests fail against the old code (verified by reverting the four
source files and re-running).

Full suite with BNGPATH set: 5093 passed, 25 skipped. Docs rebuilt clean under
-W --keep-going.

Docs

algorithms.rst described ybar as "the average of the entire data column", which is now
wrong, and the objective-function preamble did not mention that missing points are skipped at
all. Both now say what the code does, as do the two column_mean noise-source descriptions
in config_keys.rst and noise_models.rst.

Not in this PR

While confirming the above I found the same hazard in three normalizations #479 did not
reach — normalize_to_zero NaNs a whole sparse column, _subtract_baseline does too when
row 0 is unmeasured, and normalize_to_unit_scale then crashes with
ValueError: All-NaN slice encountered. Separate code, separate config keys, and the
_subtract_baseline fix needs a call on what "baseline row" means for a sparse column, so I
left it out rather than widen this. Filing separately.

…s only (#707)

AveNormSumOfSquaresObjective.evaluate precomputed each column's normalizer as
np.average(exp_data[name]) -- over the raw column, NaNs included. A NaN in an experimental
column is missing data and a first-class supported value: the exp file reader parses it
natively, docs/config.rst tells users to write it, SummationObjective.evaluate skips those
rows, and the #479 follow-up already made every Data normalization reduce past them, because
a sparse multi-observable column carries NaN wherever that observable was not measured.

But this mean is a divisor. Taken over the raw column it was NaN, so
((sim - exp) / ybar)**2 returned NaN for every present point of the column too, and the
whole objective went NaN.

Nothing downstream recovered it. evaluate gates the prediction for NaN and inf but not the
accumulated func_value, so it returned a float rather than None -- a NaN is not a failed
simulation, so it never took the failed-simulation path and was handed to the optimizer as a
real score. Every NaN comparison is False, so new_score < best_score was never true: every
parameter set was silently rejected, and the fit ran to completion reporting no improvement,
with no error and nothing in the log to say why. Trajectory.add turns the NaN into inf to
keep the heap ordered, which hides it further.

Four sites computed this same column mean and all four had it, so the fix is one shared
pybnf.data.observed_mean that averages the non-NaN entries:

  - the legacy objfunc = ave_norm_sos (pybnf/objective.py);
  - the modern objective = ave_norm_sos, which desugars to ColumnMeanSigma, whose value()
    was the same np.average -- its comment claimed the result is identical to the legacy
    path, and it was, poisoning included. Both now read Data.column_mean, so that claim is
    structural rather than a parallel implementation;
  - the PEtab export, which writes the mean as an observable's noiseFormula constant. A
    sparse problem exported the literal 'nan', which is not a usable PEtab problem -- and
    PEtab's own round trip writes 'nan' for an unmeasured point, so sparse data is exactly
    what arrives here;
  - the PEtab import, which compares a constant sigma against the column mean to tell
    ave_norm_sos from sos. NaN compares equal to nothing, so a round-tripped ave_norm_sos
    came back silently downgraded to sos.

On a dense column observed_mean is the plain mean, so every existing fit and every existing
golden value is byte-identical. A column with no measured point at all has no mean and gives
NaN; that is harmless, because every one of its rows is skipped in scoring and the normalizer
is never read. It is written out rather than using np.nanmean so that case returns quietly
instead of warning "Mean of empty slice" once per unscored column on every evaluate() of the
fit.

Tests: the NaN-in-experiment case was untested for this objfunc -- the only ave_norm_sos NaN
test put the NaN in the simulation. Added the sparse-column value for the legacy objfunc
(test_objective_funcs), for the desugared modern surface (test_objective_surface), for the
PEtab export's noiseFormula (test_petab_export), and the wholly-unmeasured-column case,
which also asserts no warning is raised. All four fail against the old code. Two unit tests
for observed_mean itself sit in test_data_class beside the #479 nan-aware normalization test.

The independent-variable column and the _SD columns get a mean computed too and never read;
that is unchanged.

Docs: algorithms.rst described ybar as "the average of the entire data column", which is now
wrong, and the objective-function preamble did not mention that missing points are skipped at
all. Both say what the code does now, as do the two column_mean noise-source descriptions in
config_keys.rst and noise_models.rst.

Signed-off-by: Bill Hlavacek <hlavacek@lanl.gov>
@wshlavacek
wshlavacek merged commit 713e013 into main Sep 17, 2026
10 of 11 checks passed
@wshlavacek
wshlavacek deleted the fix-707-ave-norm-sos-nanmean branch September 17, 2026 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ave_norm_sos's column average is NaN-poisoned by any missing data point, turning the whole objective into NaN

1 participant