information_criteria.txt reports the surviving replicate count as though it were the requested one, so a log-likelihood averaged over 3 of 10 runs is indistinguishable from one averaged over 3 of 3

[wshlavacek]

What happens

Results/information_criteria.txt reports replicates as the number of simulations that produced a usable log-likelihood, not the number that were run. A replicate that fails, or that scores nothing, is dropped by _compute_information_criteria (pybnf/algorithms/base.py, the if res is None or getattr(res, 'failed', False) or res.simdata is None: continue loop) and replicated_information_criteria then sets replicates=len(values).

Those numbers are honest. The problem is that nothing in the file says a drop happened, and the header explains replicates as though one could not have:

# replicates is how many simulations of the best fit log_likelihood is averaged
#   over. One for a deterministic model, whose simulation is exact. For a
#   stochastic model every simulation is a draw, so at the end of the fit the
#   best fit is run best_fit_replicates times and log_likelihood is the mean over
#   those runs; ...

A reader who set best_fit_replicates = 10 and sees replicates 3 is told by this header that replicates is how many times the best fit was run. It is not; it is how many of the ten runs survived. The other seven are reported only by logger.warning('%d of %d simulation(s) of the best fit produced a usable log-likelihood ...'), which never reaches the file or the console line.

Why it matters

The writer's own docstring says these criteria "rank this fit against competing models (lower is better); this is the first-class form of the AIC lesson 45 (model selection) computes by hand". That comparison is made across runs, by a reader holding two information_criteria.txt files side by side.

If model A's best fit drops seven of its ten information-criteria replicates and model B's drops none, A's log_likelihood is a mean over the three runs that worked. A failed or non-finite run is a bad outcome rather than a missing measurement, so dropping it biases the mean upward in exactly the way #720 described, and A can win the AIC comparison on a survivor mean. Neither file makes that visible; the only in-file signal is that A says replicates 3 where the reader configured 10, and the header tells them that number means something else.

This is milder than #720 and the scope below is deliberately smaller. Nothing is pinned on the trajectory here, nothing downstream consumes the value automatically, and the count in the file is not wrong -- only unexplained. It is a disclosure gap, not a wrong number.

The inconsistency #740 created

ADR-0131 aligns these two files on purpose: the mean of the per-simulation log-likelihoods "is the quantity best_fit_confirmation.txt averages, so the two files estimate the same thing for the same parameter set".

After #740 they no longer report it the same way. best_fit_confirmation.txt now carries winner_runs and winner_failed, a confirmed column, an unconfirmed line, and a console line naming the winner's failed runs. information_criteria.txt carries the survivor count alone. The two files describe the same parameter set over the same kind of replicate runs and disagree about whether a failure is worth mentioning.

Reproduction

Read-only repro, run from the repository root with
uv run --extra tests --extra petab python <file>:

import sys, os, tempfile
sys.path.insert(0, '.')
from tests.test_information_criteria_replicates import _ic_algo, LOGLIK
from tests.test_best_fit_confirmation import _ps, _FakeClient, _FakeAsCompleted
from tests.context import algorithms
algorithms.core.as_completed = _FakeAsCompleted   # synchronous dask

LOGLIK.clear()
LOGLIK[5] = [-1.0, -1.0]   # usable
LOGLIK[6] = None           # scores nothing
LOGLIK[7] = None           # scores nothing
LOGLIK[8] = [-3.0, -3.0]   # usable

algo = _ic_algo(tempfile.mkdtemp(), candidates=1, replicates=4)
algo.trajectory.add(_ps(1.0), 1.0, 'best')
ic = algo._compute_information_criteria(_ps(1.0), replicates=4, client=_FakeClient())
print('requested 4; ic.replicates =', ic.replicates)
algo._emit_information_criteria(ic)
print(open(os.path.join(algo.res_dir, 'information_criteria.txt')).read())

Observed. Four replicates requested, two usable. The log says so:

2 of 4 simulation(s) of the best fit produced a usable log-likelihood for the information criteria

The console line does not:

Information criteria (best fit): AIC=10  BIC=8.69315  AICc=n/a (n <= k+1)
  (k=1, n=2, lnL=-4, over 2 runs, lnL standard error 2)

and neither does the file, whose header above these lines says the best fit "is run
best_fit_replicates times and log_likelihood is the mean over those runs":

k	1
n	2
replicates	2
log_likelihood	-4
log_likelihood_standard_error	2
AIC	10
BIC	8.693147181
AICc	n/a (n <= k+1)

Nothing distinguishes this from a fit that asked for two replicates and got two.

Suggested scope

Disclosure only, matching what #740 did for the sibling file:

• Report how many simulations were requested alongside how many produced a usable log-likelihood -- the shape best_fit_confirmation.txt uses is winner_runs / winner_failed.
• Reword the replicates header so it stops implying the count equals best_fit_replicates.
• Put the existing log-only warning into the file, and into the console line, when the two counts differ.

Explicitly not in scope, and different from #720 on purpose: no change to the averaging, and no minimum-success threshold. #720's bar exists because a survivor mean was ranked against other candidates' means inside one run, and a candidate measured over one run could beat one measured over ten. There is no competing candidate here -- this is an estimate of the single parameter set the run already settled on -- so that argument does not transfer, and refusing to emit criteria over a low survivor count would remove information rather than add it.

Where

pybnf/algorithms/base.py, _compute_information_criteria (the drop and its log-only warning) and _emit_information_criteria (the header and the emitted keys); pybnf/objective.py, replicated_information_criteria (replicates=len(values)). Severity low, confidence high. Repro executed: True.

Follows #720 / #740 and ADR-0146, whose Consequences section records this as deliberately left out of that fix's scope.