fix(algorithms): an objective value that cannot be ordered is discarded, not passed on (#713) - #728
Merged
Merged
Conversation
…ed, not passed on (#713) Every optimizer decides by comparing scores, so a value that is not an ordinary number does not merely give a wrong answer -- it gives no answer at all, silently. A NaN is inert: nan < best is False so it never becomes the new best, and nan > threshold is False so it is never rejected as bad either. The parameter set is neither accepted nor discarded, and the fit runs to completion reporting nothing. #707 was one producer of such a score; this is why that bug was fatal rather than merely wrong, and fixing it here makes the whole class impossible rather than one instance of it. score_result is the right place because the run loop calls it, through add_to_trajectory, before it reads res.score for the min_objective check and before it hands the result to the algorithm's own got_result -- which is where all fourteen optimizers and samplers read the score. Trajectory.add already mapped NaN to inf, but only for its own heap ordering; res.score itself stayed NaN, and that is the object the optimizers read. Two things were wrong, and the placement was the more serious of them. The guard sat INSIDE the "score it here on the master" branch, so it only ever saw a result this process had just scored. A result the workers had already scored -- the default path, where an ObjectiveCalculator is scattered -- reached no check at all. The guard now runs after both paths converge. Its predicate tested res.score is None. A NaN is not None, so it did not fire. The predicate is now unusable_score, which also rejects -inf (the mirror image of NaN: it wins every comparison forever, so one degenerate evaluation pins itself as the best fit for the rest of the run) and anything not numeric at all, since a user post-processing script assigns res.score directly and is under no obligation to assign a number. +inf is deliberately NOT rejected: it is this codebase's established "discard this parameter set" sentinel, written by the #388 handler and by the worker path, and it orders correctly against every real score. Treating it as unusable would log a spurious warning on every failed simulation. The dispatch above the guard still tests None alone, and must. A NaN is a score that WAS computed, not the absence of one; re-scoring it would re-run normalize() over simdata the workers already normalized -- those transforms rewrite the column in place, so a second pass corrupts it -- on top of repeating the objective. The issue reads the two checks as one kind of guard; they are a dispatch and a validity check, and only the second should widen. The warning now names the offending value instead of asserting the simulation data contained NaNs, which was written for the None case and is not what happened when a finite simulation produces a NaN score (#707's shape exactly). core.py's worker-side guard is also None-only and is left as it is: a NaN there now travels to score_result and is caught at the chokepoint. Fixing it in place would mean moving unusable_score somewhere core.py can import from, since base.py imports core, for no change in behaviour. base.py's confirmation check (score is None or not np.isfinite(score)) keeps its own predicate: there +inf IS a failure of that replicate, which is a different question from whether a score can be ordered. Tests: test_run_loop gains TestUnusableScoreGuard -- the worker-scored path for each unusable value, the master-scored NaN (#707's shape) and None, that a usable score including +inf is returned untouched, that the value is named in the log, that a worker-scored result is not re-normalized or re-scored, and that a NaN never reaches an algorithm's got_result. Seven of the nine fail against the old code; the two that pass are the controls -- the None case it already handled, and the usable scores it must leave alone. Signed-off-by: Bill Hlavacek <hlavacek@lanl.gov>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #713. Pairs with #707/#725: that fixed one producer of a NaN score; this makes the whole class harmless.
Why a NaN score is worse than a huge one
Every optimizer decides by comparing scores. A NaN is inert, not wrong:
nan < bestis False so it never becomes the new best, andnan > thresholdis False so it is never rejected as bad either. The parameter set is neither accepted nor discarded, and the fit runs to completion reporting nothing.Trajectory.addalready mapped NaN toinf, but only for its own heap ordering.res.scoreitself stayed NaN — and that is the object all fourteen optimizers and samplers read.Two defects, and the placement was the worse one
The guard was in the wrong place. It sat inside the "score it here on the master" branch, so it only ever saw a result this process had just scored. A result the workers already scored — the default path, where an
ObjectiveCalculatoris scattered — reached no check at all. It now runs after both paths converge.The predicate was too narrow. It tested
res.score is None; a NaN is notNone. It is nowunusable_score, which also rejects:-inf— the mirror image of NaN. It wins every comparison forever, so one degenerate evaluation pins itself as the best fit for the rest of the run.res.scoredirectly and is under no obligation to assign a number.+infis deliberately not rejected: it is this codebase's established "discard this parameter set" sentinel, written by the #388 handler and the worker path, and it orders correctly against every real score. Treating it as unusable would log a spurious warning on every failed simulation.One correction to the issue
The issue reads the two checks at
base.py:855and:874as the same kind of guard and implies both should widen. They aren't: the first is a dispatch ("has this been scored yet?"), the second a validity check. Only the second should widen.The dispatch must keep testing
Nonealone. A NaN is a score that was computed, not the absence of one — re-scoring it would re-runnormalize()over simdata the workers already normalized, and those transforms rewrite the column in place, so a second pass corrupts it, on top of repeating the objective. There's a test pinning that.Why
score_resultis the right chokepointI verified the ordering rather than taking it on faith. The run loop does:
So the score an algorithm sees is always orderable.
Sites deliberately left alone
core.py's worker-side guard is alsoNone-only. A NaN there now travels toscore_resultand is caught at the chokepoint. Fixing it in place would mean movingunusable_scoresomewherecore.pycan import from —base.pyimportscore, not the reverse — for no change in behaviour.base.py's confirmation check (score is None or not np.isfinite(score)) keeps its own predicate. There+infis a failure of that replicate, which is a different question from whether a score can be ordered.Tests
test_run_loop.pygainsTestUnusableScoreGuard: the worker-scored path per unusable value; the master-scored NaN (#707's exact shape) andNone; that a usable score including+infis returned untouched; that the value is named in the log; that a worker-scored result is not re-normalized or re-scored; and that a NaN never reaches an algorithm'sgot_result.7 of the 9 fail against the old code. The 2 that pass are the controls — the
Nonecase it already handled, and the usable scores it must leave alone.Full suite with
BNGPATHset: 5120 passed, 25 skipped.uvx ruff@0.15.14 check .clean.No docs change: this is an internal guard, and no documentation quotes the log strings I replaced.