Skip to content

fix(algorithms): an objective value that cannot be ordered is discarded, not passed on (#713) - #728

Merged
wshlavacek merged 1 commit into
mainfrom
fix-713-nan-score-guard
Sep 17, 2026
Merged

wshlavacek merged 1 commit into
mainfrom
fix-713-nan-score-guard

Conversation

@wshlavacek

Copy link
Copy Markdown
Collaborator

Closes #713. Pairs with #707/#725: that fixed one producer of a NaN score; this makes the whole class harmless.

Why a NaN score is worse than a huge one

Every optimizer decides by comparing scores. A NaN is inert, not wrong: nan < best is False so it never becomes the new best, and nan > threshold is False so it is never rejected as bad either. The parameter set is neither accepted nor discarded, and the fit runs to completion reporting nothing.

Trajectory.add already mapped NaN to inf, but only for its own heap ordering. res.score itself stayed NaN — and that is the object all fourteen optimizers and samplers read.

Two defects, and the placement was the worse one

The guard was in the wrong place. It sat inside the "score it here on the master" branch, so it only ever saw a result this process had just scored. A result the workers already scored — the default path, where an ObjectiveCalculator is scattered — reached no check at all. It now runs after both paths converge.

The predicate was too narrow. It tested res.score is None; a NaN is not None. It is now unusable_score, which also rejects:

  • -inf — the mirror image of NaN. It wins every comparison forever, so one degenerate evaluation pins itself as the best fit for the rest of the run.
  • anything not numeric — a user post-processing script assigns res.score directly and is under no obligation to assign a number.

+inf is deliberately not rejected: it is this codebase's established "discard this parameter set" sentinel, written by the #388 handler and the worker path, and it orders correctly against every real score. Treating it as unusable would log a spurious warning on every failed simulation.

One correction to the issue

The issue reads the two checks at base.py:855 and :874 as the same kind of guard and implies both should widen. They aren't: the first is a dispatch ("has this been scored yet?"), the second a validity check. Only the second should widen.

The dispatch must keep testing None alone. A NaN is a score that was computed, not the absence of one — re-scoring it would re-run normalize() over simdata the workers already normalized, and those transforms rewrite the column in place, so a second pass corrupts it, on top of repeating the objective. There's a test pinning that.

Why score_result is the right chokepoint

I verified the ordering rather than taking it on faith. The run loop does:

self.add_to_trajectory(res)          -> score_result(res)   # sanitizes here
if res.score < min_objective: ...
response = self.got_result(res)                             # every optimizer reads it here

So the score an algorithm sees is always orderable.

Sites deliberately left alone

  • core.py's worker-side guard is also None-only. A NaN there now travels to score_result and is caught at the chokepoint. Fixing it in place would mean moving unusable_score somewhere core.py can import from — base.py imports core, not the reverse — for no change in behaviour.
  • base.py's confirmation check (score is None or not np.isfinite(score)) keeps its own predicate. There +inf is a failure of that replicate, which is a different question from whether a score can be ordered.

Tests

test_run_loop.py gains TestUnusableScoreGuard: the worker-scored path per unusable value; the master-scored NaN (#707's exact shape) and None; that a usable score including +inf is returned untouched; that the value is named in the log; that a worker-scored result is not re-normalized or re-scored; and that a NaN never reaches an algorithm's got_result.

7 of the 9 fail against the old code. The 2 that pass are the controls — the None case it already handled, and the usable scores it must leave alone.

Full suite with BNGPATH set: 5120 passed, 25 skipped. uvx ruff@0.15.14 check . clean.

No docs change: this is an internal guard, and no documentation quotes the log strings I replaced.

…ed, not passed on (#713)

Every optimizer decides by comparing scores, so a value that is not an ordinary number does
not merely give a wrong answer -- it gives no answer at all, silently. A NaN is inert:
nan < best is False so it never becomes the new best, and nan > threshold is False so it is
never rejected as bad either. The parameter set is neither accepted nor discarded, and the
fit runs to completion reporting nothing. #707 was one producer of such a score; this is why
that bug was fatal rather than merely wrong, and fixing it here makes the whole class
impossible rather than one instance of it.

score_result is the right place because the run loop calls it, through add_to_trajectory,
before it reads res.score for the min_objective check and before it hands the result to the
algorithm's own got_result -- which is where all fourteen optimizers and samplers read the
score. Trajectory.add already mapped NaN to inf, but only for its own heap ordering; res.score
itself stayed NaN, and that is the object the optimizers read.

Two things were wrong, and the placement was the more serious of them.

The guard sat INSIDE the "score it here on the master" branch, so it only ever saw a result
this process had just scored. A result the workers had already scored -- the default path,
where an ObjectiveCalculator is scattered -- reached no check at all. The guard now runs after
both paths converge.

Its predicate tested res.score is None. A NaN is not None, so it did not fire. The predicate
is now unusable_score, which also rejects -inf (the mirror image of NaN: it wins every
comparison forever, so one degenerate evaluation pins itself as the best fit for the rest of
the run) and anything not numeric at all, since a user post-processing script assigns
res.score directly and is under no obligation to assign a number. +inf is deliberately NOT
rejected: it is this codebase's established "discard this parameter set" sentinel, written by
the #388 handler and by the worker path, and it orders correctly against every real score.
Treating it as unusable would log a spurious warning on every failed simulation.

The dispatch above the guard still tests None alone, and must. A NaN is a score that WAS
computed, not the absence of one; re-scoring it would re-run normalize() over simdata the
workers already normalized -- those transforms rewrite the column in place, so a second pass
corrupts it -- on top of repeating the objective. The issue reads the two checks as one kind
of guard; they are a dispatch and a validity check, and only the second should widen.

The warning now names the offending value instead of asserting the simulation data contained
NaNs, which was written for the None case and is not what happened when a finite simulation
produces a NaN score (#707's shape exactly).

core.py's worker-side guard is also None-only and is left as it is: a NaN there now travels to
score_result and is caught at the chokepoint. Fixing it in place would mean moving
unusable_score somewhere core.py can import from, since base.py imports core, for no change in
behaviour. base.py's confirmation check (score is None or not np.isfinite(score)) keeps its
own predicate: there +inf IS a failure of that replicate, which is a different question from
whether a score can be ordered.

Tests: test_run_loop gains TestUnusableScoreGuard -- the worker-scored path for each unusable
value, the master-scored NaN (#707's shape) and None, that a usable score including +inf is
returned untouched, that the value is named in the log, that a worker-scored result is not
re-normalized or re-scored, and that a NaN never reaches an algorithm's got_result. Seven of
the nine fail against the old code; the two that pass are the controls -- the None case it
already handled, and the usable scores it must leave alone.

Signed-off-by: Bill Hlavacek <hlavacek@lanl.gov>
@wshlavacek
wshlavacek merged commit f3bd79f into main Sep 17, 2026
7 checks passed
@wshlavacek
wshlavacek deleted the fix-713-nan-score-guard branch September 17, 2026 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

score_result's NaN guard tests only for None, so a NaN objective escapes into every optimizer (the correct predicate already exists at base.py:1838)

1 participant