fix(objective): ave_norm_sos's normalizer averages the measured points only (#707) - #725
Merged
Merged
Conversation
…s only (#707) AveNormSumOfSquaresObjective.evaluate precomputed each column's normalizer as np.average(exp_data[name]) -- over the raw column, NaNs included. A NaN in an experimental column is missing data and a first-class supported value: the exp file reader parses it natively, docs/config.rst tells users to write it, SummationObjective.evaluate skips those rows, and the #479 follow-up already made every Data normalization reduce past them, because a sparse multi-observable column carries NaN wherever that observable was not measured. But this mean is a divisor. Taken over the raw column it was NaN, so ((sim - exp) / ybar)**2 returned NaN for every present point of the column too, and the whole objective went NaN. Nothing downstream recovered it. evaluate gates the prediction for NaN and inf but not the accumulated func_value, so it returned a float rather than None -- a NaN is not a failed simulation, so it never took the failed-simulation path and was handed to the optimizer as a real score. Every NaN comparison is False, so new_score < best_score was never true: every parameter set was silently rejected, and the fit ran to completion reporting no improvement, with no error and nothing in the log to say why. Trajectory.add turns the NaN into inf to keep the heap ordered, which hides it further. Four sites computed this same column mean and all four had it, so the fix is one shared pybnf.data.observed_mean that averages the non-NaN entries: - the legacy objfunc = ave_norm_sos (pybnf/objective.py); - the modern objective = ave_norm_sos, which desugars to ColumnMeanSigma, whose value() was the same np.average -- its comment claimed the result is identical to the legacy path, and it was, poisoning included. Both now read Data.column_mean, so that claim is structural rather than a parallel implementation; - the PEtab export, which writes the mean as an observable's noiseFormula constant. A sparse problem exported the literal 'nan', which is not a usable PEtab problem -- and PEtab's own round trip writes 'nan' for an unmeasured point, so sparse data is exactly what arrives here; - the PEtab import, which compares a constant sigma against the column mean to tell ave_norm_sos from sos. NaN compares equal to nothing, so a round-tripped ave_norm_sos came back silently downgraded to sos. On a dense column observed_mean is the plain mean, so every existing fit and every existing golden value is byte-identical. A column with no measured point at all has no mean and gives NaN; that is harmless, because every one of its rows is skipped in scoring and the normalizer is never read. It is written out rather than using np.nanmean so that case returns quietly instead of warning "Mean of empty slice" once per unscored column on every evaluate() of the fit. Tests: the NaN-in-experiment case was untested for this objfunc -- the only ave_norm_sos NaN test put the NaN in the simulation. Added the sparse-column value for the legacy objfunc (test_objective_funcs), for the desugared modern surface (test_objective_surface), for the PEtab export's noiseFormula (test_petab_export), and the wholly-unmeasured-column case, which also asserts no warning is raised. All four fail against the old code. Two unit tests for observed_mean itself sit in test_data_class beside the #479 nan-aware normalization test. The independent-variable column and the _SD columns get a mean computed too and never read; that is unchanged. Docs: algorithms.rst described ybar as "the average of the entire data column", which is now wrong, and the objective-function preamble did not mention that missing points are skipped at all. Both say what the code does now, as do the two column_mean noise-source descriptions in config_keys.rst and noise_models.rst. Signed-off-by: Bill Hlavacek <hlavacek@lanl.gov>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #707.
The defect
AveNormSumOfSquaresObjective.evaluateprecomputed each column's normalizer withnp.averageover the raw column, NaNs included. A NaN in an experimental column ismissing data and a first-class supported value — the reader parses it natively,
docs/config.rsttells users to write it,SummationObjective.evaluateskips those rows,and #479 already made every
Datanormalization reduce past them, because a sparsemulti-observable column carries NaN wherever that observable was not measured.
But this mean is a divisor. Over the raw column it was NaN, so
((sim - exp) / ybar)**2returned NaN for every present point of the column too, and the whole objective went NaN.
Nothing downstream recovered it.
evaluategates the prediction for NaN/inf but not theaccumulated
func_value, so it returned a float rather thanNone— a NaN is not a failedsimulation, so it never took the failed-simulation path and reached the optimizer as a real
score. Every NaN comparison is False, so
new_score < best_scorewas never true: everyparameter set was silently rejected, and the fit ran to completion reporting no improvement,
with no error and nothing in the log to say why.
Scope: four sites, not two
The issue named two. Four computed this same column mean, and all four had it:
pybnf/objective.py— legacyobjfunc = ave_norm_sospybnf/noise/source.py—ColumnMeanSigma, i.e. modernobjective = ave_norm_sospybnf/petab/export.pynanas an observable'snoiseFormula— not a usable PEtab problempybnf/petab/import_.pyave_norm_sosfromsos; NaN compares equal to nothing, so a round-trippedave_norm_soscame back silently downgraded tososThe two PEtab sites are not hypothetical: PEtab's own round-trip writes
nanfor anunmeasured point (
import_.py:1219), so a sparse column is exactly what arrives there.ColumnMeanSigma.value's comment claimed its result was identical to the legacy path. Itwas — poisoning included. All four now read one
pybnf.data.observed_mean, so that claim isstructural rather than two parallel implementations that happened to agree.
Behaviour
On a dense column
observed_meanis the plain mean, so every existing fit and golden valueis byte-identical. A column with no measured point has no mean and gives NaN; harmless,
since every one of its rows is skipped in scoring and the normalizer is never read. It is
written out rather than using
np.nanmeanso that case returns quietly instead of warningMean of empty sliceonce per unscored column on everyevaluate()of the fit.Tests
The NaN-in-experiment case was untested for this objfunc — the one existing
ave_norm_sosNaN test puts the NaN in the simulation. Added:
test_objective_funcs.py— the sparse-column value for the legacy objfunc, and thewholly-unmeasured-column case (which also asserts no warning is raised)
test_objective_surface.py— the same for the desugared modern surfacetest_petab_export.py— the exportednoiseFormulaon a sparse problemtest_data_class.py— two unit tests forobserved_mean, beside Feature request: composable floor-normalization + analytic per-series scaling for relative / arbitrary-unit data #479's nan-aware testAll four behavioural tests fail against the old code (verified by reverting the four
source files and re-running).
Full suite with
BNGPATHset: 5093 passed, 25 skipped. Docs rebuilt clean under-W --keep-going.Docs
algorithms.rstdescribed ybar as "the average of the entire data column", which is nowwrong, and the objective-function preamble did not mention that missing points are skipped at
all. Both now say what the code does, as do the two
column_meannoise-source descriptionsin
config_keys.rstandnoise_models.rst.Not in this PR
While confirming the above I found the same hazard in three normalizations #479 did not
reach —
normalize_to_zeroNaNs a whole sparse column,_subtract_baselinedoes too whenrow 0 is unmeasured, and
normalize_to_unit_scalethen crashes withValueError: All-NaN slice encountered. Separate code, separate config keys, and the_subtract_baselinefix needs a call on what "baseline row" means for a sparse column, so Ileft it out rather than widen this. Filing separately.