Skip to content

fix(information-criteria): report how many simulations of the best fit were run beside how many produced a usable log-likelihood (#741) - #742

Merged
wshlavacek merged 1 commit into
mainfrom
fix/741-information-criteria-replicate-disclosure
Sep 17, 2026
Merged

wshlavacek merged 1 commit into
mainfrom
fix/741-information-criteria-replicate-disclosure

Conversation

@wshlavacek

Copy link
Copy Markdown
Collaborator

Fixes #741.

The defect

A simulation that fails, scores nothing, or scores a different number of points from the rest is left out of the mean behind AIC, BIC and AICc, and replicated_information_criteria set replicates to the surviving count. That number was honest — but nothing in the file said a run had been lost, and the header explained replicates as though none could be:

the best fit is run best_fit_replicates times and log_likelihood is the mean over those runs

So a log-likelihood averaged over 3 of 10 runs read exactly like one averaged over 3 of 3. The %d of %d simulation(s) warning went to the log alone, and the console's only mention of the count rides on a clause that appears once two runs have produced a value — so the case where nine of ten were lost was the case it hid.

This matters where the criteria are used. _emit_information_criteria's own docstring says they "rank this fit against competing models"; that comparison is made by a reader holding two of these files side by side. A model whose best fit lost seven of its ten runs is scored on a mean over the three that worked, optimistic in the way #720 described, and can win the AIC comparison on it.

The fix

InformationCriteria carries replicates_requested; replicated_information_criteria takes requested; _compute_information_criteria passes len(jobs).

Before, with four simulations run and two usable:

k	1
n	2
replicates	2
log_likelihood	-4

After:

#
# 2 of the 4 simulations of the best fit produced no usable log-likelihood,
#   so log_likelihood below is the mean over the 2 that did. A run that
#   produces nothing is a bad outcome rather than a missing measurement, so
#   that mean is optimistic by however much the lost runs would have pulled
#   it down. Weigh it against another model's accordingly, and look into why
#   the simulations are failing. The log says what went wrong with each.
k	1
n	2
replicates_requested	4
replicates	2
log_likelihood	-4

The note sits above the numbers so it is read before them. The console prints the same on its own line rather than folded into the standard-error clause, so a single surviving run still reports what was lost:

Information criteria (best fit): AIC=6  BIC=4.69315  AICc=n/a (n <= k+1)  (k=1, n=2, lnL=-2)
  9 of the 10 simulations of the best fit produced no usable log-likelihood, so that is
  the mean over the 1 that did and is optimistic by however much the rest would have
  pulled it down.

A run that lost nothing says nothing, on the console and in the file. A criteria object built without a request count is written exactly as before.

Scope

The averaging is unchanged and no minimum-success threshold was added, unlike #720's confirmation ranking. That bar exists because candidate means were ranked against each other inside one run and a candidate measured over one run could beat one measured over ten. This file holds one parameter set the run has already settled on, so there is nothing for a survivor mean to beat unfairly, and refusing to emit criteria over a low count would remove information rather than add it.

ADRs

Testing

  • Seven new tests: the requested count riding along and defaulting to the number of values; the file's new key and loss note; the console's loss line; the single-survivor case where the old over N runs clause is absent; a run that lost nothing staying silent in both places; a criteria object with no request count still being written; and the whole thing end to end through _compute_information_criteria with a synchronous dask client.
  • Two existing tests updated: the pinned key-order assertion gains replicates_requested, and two drop tests now pin the requested count too.
  • Full suite with BNGPATH set: 5183 passed, 25 skipped, 0 failed (5176 before this branch).
  • sphinx-build -W --keep-going and ruff check . both clean.

…t were run beside how many produced a usable log-likelihood (#741)

A simulation that fails, that scores nothing, or that scores a different
number of points from the rest is left out of the mean behind AIC, BIC and
AICc, and replicated_information_criteria set `replicates` to the surviving
count. That number was honest. Nothing in the file said a run had been lost,
and the header explained `replicates` as though none could be: "the best fit
is run best_fit_replicates times and log_likelihood is the mean over those
runs". So a log-likelihood averaged over 3 of 10 runs read exactly like one
averaged over 3 of 3.

The `%d of %d simulation(s)` warning went to the log alone. The console's only
mention of the count rides on a clause that appears once two runs have produced
a value, so the case where nine of ten were lost was the case it hid.

This matters where the criteria are used. _emit_information_criteria's own
docstring says they "rank this fit against competing models"; that comparison
is made by a reader holding two of these files side by side. A model whose
best fit lost seven of its ten runs is scored on a mean over the three that
worked, optimistic in the way #720 described, and can win the AIC comparison
on it.

InformationCriteria now carries replicates_requested and
replicated_information_criteria takes `requested`; _compute_information_criteria
passes len(jobs). The file carries a replicates_requested line beside
replicates and, when they differ, says above the numbers how many runs
produced nothing and that the mean is optimistic by however much they would
have pulled it down. The console says the same, on its own line rather than
folded into the standard-error clause, so a single surviving run still reports
what was lost. A criteria object built without a request count is written
exactly as before, and a run that lost nothing says nothing.

The averaging is unchanged and no minimum-success threshold was added, unlike
#720's confirmation ranking. That bar exists because candidate means were
ranked against each other inside one run, and a candidate measured over one run
could beat one measured over ten. This file holds one parameter set the run has
already settled on, so there is nothing for a survivor mean to beat unfairly,
and refusing to emit criteria over a low count would remove information rather
than add it.

ADR-0131 is amended rather than rewritten: its file example and its sentence
about dropped runs describe the file as it shipped, with a note pointing here.
ADR-0146's Consequences entry, which recorded this as deliberately out of
scope for #720, is corrected -- it judged the averaging and missed that the
sibling file did not even say it had dropped anything, a gap #740 itself
widened.

Signed-off-by: Bill Hlavacek <bill.hlavacek@gmail.com>
@wshlavacek
wshlavacek merged commit 70575cb into main Sep 17, 2026
7 checks passed
@wshlavacek
wshlavacek deleted the fix/741-information-criteria-replicate-disclosure branch September 17, 2026 22:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant