fix(best-fit-confirmation): rank a candidate on all of its replicate runs, so one that fails most of them is not pinned as the run's best fit (#720)

[wshlavacek]

Fixes #720.

The defect

The best-fit confirmation stage (#659) exists to stop a stochastic fit reporting whichever parameter set got a lucky simulation: it runs the search's top best_fit_candidates parameter sets best_fit_replicates more times each and reports the one with the best average.

A replicate that fails, or that returns a non-finite objective value, produces no number to average. mean_objective averaged only the runs that did — a survivor-only mean over a sample size that varied per candidate from 1 to best_fit_replicates. The failure count travelled on the row and was printed in a column, but it entered no comparison: ranked keyed on that mean alone and winner disqualified a candidate only when every one of its runs had failed.

So a parameter set that failed nine runs of ten and survived one was scored on a single simulation again — the max-of-lucky-draws estimate, now with the unlucky draws deleted rather than averaged in — and it beat a parameter set that returned a slightly worse value in ten runs of ten. _emit_best_fit_confirmation pins the winner on the trajectory by design, so the saved simulations, the best-fit BNGL, the information criteria, a refine's start point and a bootstrap replicate's answer then all described the failure-prone set.

Run end to end on the test harness before this change:

winner  flaky
# rank  name      mean_objective  standard_error  std_deviation  runs  failed  search_objective
1       flaky     5               n/a             n/a            1     9       5.2
2       reliable  5.1             0               0              10    0       5.05

and on the console, with no mention of the nine failures, the flatly false line "the one the search liked best does worse when it is run again".

The fix

A candidate is now confirmed, and eligible to win, only when more than half of the runs that came back produced a usable value.

Half is not a round number picked for convenience. A failure is known to be worse than every finite value, so it is exactly the condition for the middle of all of a candidate's runs to be a real number. Above the bar the survivor mean averages most of what happened; below it, it describes the minority that worked, which is the lucky draw the stage exists to remove. The bar is written against the runs that came back rather than against best_fit_replicates, because a cancelled replicate is the cluster's doing and not the parameter set's.

Unconfirmed candidates are ranked below every confirmed one whatever their averages say. When none clears the bar the stage pins nothing and the search's own pick stands, rather than the best of a bad lot.

Nothing is hidden: mean_objective still reports the survivor average, the table gains a confirmed column and an unconfirmed line naming the demoted candidates, and winner_runs / winner_failed put the winner's sample size in the file.

After:

winner  reliable
winner_runs     10
winner_failed   0
unconfirmed     flaky
# rank  name      mean_objective  standard_error  std_deviation  runs  failed  confirmed  search_objective
1       reliable  5.1             0               0              10    0       yes        5.05
2       flaky     5               n/a             n/a            1     9       no         5.2

A candidate that loses a minority of its runs is unaffected: it is confirmed, ranked on its average exactly as before, and can win — with a note saying how many runs it lost.

The silence

The other half of the issue. console_lines never mentioned failures, and with a single survivor standard_error is None, so the documented "raise best_fit_replicates if the standard errors overlap" guidance could not flag it either. The console now reports the winner's failed runs and how many candidates were not confirmed, and where it used to assert flatly that the search's pick "does worse when it is run again" — the opposite of the truth when that candidate had the better average and lost on reliability — it now says which of the two things actually happened.

Alternatives

Recorded in ADR-0146, with why each was rejected: counting any failure as infinite (too harsh for a transient cluster timeout, and it contradicts behaviour pinned in a test since #659), ranking by failure count first (lexicographic, so one failure would outweigh any objective difference), a multiplicative penalty (PyBNF objective values are not sign-constrained — a likelihood objective is routinely negative), a configuration key, falling back to the best unconfirmed candidate, and ranking on the censored median itself — the estimator the argument actually points at, rejected for blast radius since it would change the reported value of every stochastic fit including those with no failures at all.

Testing

• Eleven new tests: the bar's arithmetic at 10/10, 6/10, 5/10, 1/10, 2/3, 1/3, 1/1 and 0; the issue's case at the arithmetic layer and end to end through _confirm_best_fit with a synchronous dask client; no winner when every candidate is too unreliable; the report's new columns and lines; and both readings of the search-rank sentence, so the true one is still printed when the search's pick really did run worse.
• The full suite was run on this branch and on main in this environment and the failing-test sets are identical (334 pre-existing failures, all simulation tests needing a built BioNetGen that is not installed here); this branch adds 11 passing tests and changes nothing else.
• sphinx-build -W --keep-going and ruff check . both clean.

[wshlavacek]

Verification update: with BNGPATH pointing at a real BioNetGen 2.9.3 install the full suite on this branch is 5176 passed, 25 skipped, 0 failed (4m12s at -n 4). That supersedes the failing-set diff described above, which was run in a shell where BNGPATH was unset and 334 simulation tests could not run at all.