fix(information-criteria): report how many simulations of the best fit were run beside how many produced a usable log-likelihood (#741) - #742
Merged
wshlavacek merged 1 commit intoSep 17, 2026
Conversation
…t were run beside how many produced a usable log-likelihood (#741) A simulation that fails, that scores nothing, or that scores a different number of points from the rest is left out of the mean behind AIC, BIC and AICc, and replicated_information_criteria set `replicates` to the surviving count. That number was honest. Nothing in the file said a run had been lost, and the header explained `replicates` as though none could be: "the best fit is run best_fit_replicates times and log_likelihood is the mean over those runs". So a log-likelihood averaged over 3 of 10 runs read exactly like one averaged over 3 of 3. The `%d of %d simulation(s)` warning went to the log alone. The console's only mention of the count rides on a clause that appears once two runs have produced a value, so the case where nine of ten were lost was the case it hid. This matters where the criteria are used. _emit_information_criteria's own docstring says they "rank this fit against competing models"; that comparison is made by a reader holding two of these files side by side. A model whose best fit lost seven of its ten runs is scored on a mean over the three that worked, optimistic in the way #720 described, and can win the AIC comparison on it. InformationCriteria now carries replicates_requested and replicated_information_criteria takes `requested`; _compute_information_criteria passes len(jobs). The file carries a replicates_requested line beside replicates and, when they differ, says above the numbers how many runs produced nothing and that the mean is optimistic by however much they would have pulled it down. The console says the same, on its own line rather than folded into the standard-error clause, so a single surviving run still reports what was lost. A criteria object built without a request count is written exactly as before, and a run that lost nothing says nothing. The averaging is unchanged and no minimum-success threshold was added, unlike #720's confirmation ranking. That bar exists because candidate means were ranked against each other inside one run, and a candidate measured over one run could beat one measured over ten. This file holds one parameter set the run has already settled on, so there is nothing for a survivor mean to beat unfairly, and refusing to emit criteria over a low count would remove information rather than add it. ADR-0131 is amended rather than rewritten: its file example and its sentence about dropped runs describe the file as it shipped, with a note pointing here. ADR-0146's Consequences entry, which recorded this as deliberately out of scope for #720, is corrected -- it judged the averaging and missed that the sibling file did not even say it had dropped anything, a gap #740 itself widened. Signed-off-by: Bill Hlavacek <bill.hlavacek@gmail.com>
wshlavacek
deleted the
fix/741-information-criteria-replicate-disclosure
branch
September 17, 2026 22:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #741.
The defect
A simulation that fails, scores nothing, or scores a different number of points from the rest is left out of the mean behind AIC, BIC and AICc, and
replicated_information_criteriasetreplicatesto the surviving count. That number was honest — but nothing in the file said a run had been lost, and the header explainedreplicatesas though none could be:So a log-likelihood averaged over 3 of 10 runs read exactly like one averaged over 3 of 3. The
%d of %d simulation(s)warning went to the log alone, and the console's only mention of the count rides on a clause that appears once two runs have produced a value — so the case where nine of ten were lost was the case it hid.This matters where the criteria are used.
_emit_information_criteria's own docstring says they "rank this fit against competing models"; that comparison is made by a reader holding two of these files side by side. A model whose best fit lost seven of its ten runs is scored on a mean over the three that worked, optimistic in the way #720 described, and can win the AIC comparison on it.The fix
InformationCriteriacarriesreplicates_requested;replicated_information_criteriatakesrequested;_compute_information_criteriapasseslen(jobs).Before, with four simulations run and two usable:
After:
The note sits above the numbers so it is read before them. The console prints the same on its own line rather than folded into the standard-error clause, so a single surviving run still reports what was lost:
A run that lost nothing says nothing, on the console and in the file. A criteria object built without a request count is written exactly as before.
Scope
The averaging is unchanged and no minimum-success threshold was added, unlike #720's confirmation ranking. That bar exists because candidate means were ranked against each other inside one run and a candidate measured over one run could beat one measured over ten. This file holds one parameter set the run has already settled on, so there is nothing for a survivor mean to beat unfairly, and refusing to emit criteria over a low count would remove information rather than add it.
ADRs
best_fit_confirmation.txtwinner_runs/winner_failedwhile this file kept the survivor count alone — in two files ADR-0131 deliberately aligns.Testing
over N runsclause is absent; a run that lost nothing staying silent in both places; a criteria object with no request count still being written; and the whole thing end to end through_compute_information_criteriawith a synchronous dask client.replicates_requested, and two drop tests now pin the requested count too.BNGPATHset: 5183 passed, 25 skipped, 0 failed (5176 before this branch).sphinx-build -W --keep-goingandruff check .both clean.