test(bmdb): accept the three changed BMDB results - #2078
Merged
Merged
Conversation
The nightly has failed every night since 2026-08-31. These are the three
deviations in the 09-07 run:
BIOMD0000000547 SOLVER_FAILURE -> MATH_GENERATION_FAILURE
BIOMD0000000613 DIVIDE_BY_ZERO -> MATH_GENERATION_FAILURE
BIOMD0000001065 SOLVER_FAILURE -> PASSED
None of them is a regression: no model went from passing to failing, and the
totals moved the right way, 914 -> 915 passes and 147 -> 146 failures.
Verified both directions against that night's own exec_summary.ndjson rather
than reasoning about it - the old baseline reproduces the failure locally
("FAIL: 3 changed", exit 1) and the new one passes ("PASS: no changes against
the documented baseline", exit 0).
Only the three rows are re-serialized; the other 1179 are untouched byte for
byte, so the diff is 3 lines.
WHAT THIS DOES NOT FIX. These cases are non-deterministic, and accepting a
baseline does not make them stop moving. Over the last seven nightlies four
different models deviated, in rotation:
1065 deviated (PASSED) on six of seven nights; matched its FAIL baseline
once, on 09-06
547 MATH_GENERATION_FAILURE on 09-04/05/06/07, SOLVER_FAILURE before that
613 MATH_GENERATION_FAILURE on 09-03/05/06/07, DIVIDE_BY_ZERO on 09-04
457 flipped PASSED -> SOLVER_FAILURE on 09-05 only, and came back
The recorded failures say why for two of them. 1065 fails on
"pow(u,v) and u=-0.000000<0" with EN_0_0 = -0.000000, and 547 on
"At t = 8.09814e-18, mxstep steps taken" - both are values sitting on the edge
of zero, where the sign of the last bit decides the outcome.
613 is the one worth looking at, because it is not numerical. It alternates
between "divide by zero '(M_initConc / OC_initConc)'" raised while writing the
solver input, and "Initial condition for variable 'M' references variable 'L'"
raised during math generation - the same initial condition, surfacing two
phases apart. That reads as ordering-dependent math generation rather than
arithmetic. Filed separately.
So expect this gate to fail again on a night when one of these lands the other
way; that is the gate working, not a new problem.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019HAnpFxkzf9LmxBayDSANf
bioModelsNetInfo.xml carries a Supported flag per BioModels entry, and BioModelsNetInfoTest keeps it honest against the nightly baseline. Accepting the new failure types moved two of them: BIOMD0000000547 Supported true -> false BIOMD0000000613 Supported true -> false That is the correct consequence rather than a side effect. Supported means VCell can IMPORT the model, so the test derives it from whether the recorded failure is import-blocking. SOLVER_FAILURE and DIVIDE_BY_ZERO are not - the model imports and then does not solve - while MATH_GENERATION_FAILURE is, and that is what both of these now report. BIOMD0000001065 needed no change for the same reason: it moved FAIL -> PASS, and SOLVER_FAILURE was already non-blocking, so its flag was true either way. Regenerated with the command the test itself prints, so the file is rewritten byte for byte apart from the two attributes; it is CRLF and stays that way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019HAnpFxkzf9LmxBayDSANf
Comments only; no behaviour change.
There are two files here and they are easy to mistake for one, because they
overlap in vocabulary and both talk about which BioModels archives VCell can
handle. They answer different questions:
test_cases.ndjson the nightly's expected-results baseline. Source of
truth for what each case did last time we accepted it,
and so both a regression gate and a compliance record.
bioModelsNetInfo.xml a prediction, shipped with the client and read at
runtime, of which archives the BMDB tab expects to
OPEN. Derived from the first; not a source of truth.
I conflated them while accepting a baseline change, argued the first one's
semantics at the second one's file, and had to be corrected. So the
definitions now live on the data models rather than in anyone's memory:
OmexTestCase, OmexExecSummary, FailureType, BioModelsNetModelInfo, the two
XML attribute constants, and a header comment in the XML itself.
Three things worth writing down that the code does not say anywhere:
- The XML predicts OPENING, not running. The tab uses the ordinary
document-open path and never simulates, so ~70 archives that import cleanly
and then fail in the solver are correctly still Supported="true". Reading
the flag as "will run" makes those look like bugs.
- FailureType mixes three kinds of thing with only the naming to tell them
apart: capability statements (UNSUPPORTED_*), defects, and operational
states. Nothing enforces it, which is why BioModelsNetInfoTest carries its
own explicit set. The enum comment now asks that new values put the kind in
the name, since an UNSUPPORTED_ prefix reads as a promise about scope.
- Status.SKIP already means two different things - excluded because too slow,
and not applicable because we do not support the feature - distinguishable
only by the description text.
Also records the two dead attributes rather than leaving them to be
rediscovered: exception= is read by nothing, which is why the tab can say a
model is not compatible but not why; Slow= is read, but sits on three rows
that are the first three of the eleven the nightly excludes for the same
reason, never reconciled.
One trap found writing this: "--" is illegal inside an XML comment, and the
client parses this file with JDOM at runtime. BioModelsNetInfoTest did not
catch it, because it is line-based rather than an XML parse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019HAnpFxkzf9LmxBayDSANf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The BMDB nightly has failed every night since 2026-08-31. This accepts the three deviations
in the 09-07 run (
34094676265):BIOMD0000000547SOLVER_FAILUREMATH_GENERATION_FAILUREBIOMD0000000613DIVIDE_BY_ZEROMATH_GENERATION_FAILUREBIOMD0000001065SOLVER_FAILURENone is a regression. No model went from passing to failing; the two that changed were
already failing and now fail at an earlier phase, and one that was failing now passes.
Totals moved the right way: 914 → 915 passes, 147 → 146 failures.
Verified, both directions
Against that night's own
exec_summary.ndjsonrather than by reasoning about it:FAIL: 3 changed, exit 1 — reproduces the nightly failure locallyPASS: no changes against the documented baseline, exit 0Only the three rows are re-serialized; the other 1179 are untouched byte for byte, so the
diff is three lines.
A second file had to move with it
bioModelsNetInfo.xmlcarries aSupportedflag per BioModels entry, andBioModelsNetInfoTestkeeps it honest against this baseline — it failed CI on the firstcommit, which is the check doing its job. Two flags moved:
That is the right consequence, not a side effect.
Supportedmeans VCell can import themodel, so the test derives it from whether the recorded failure is import-blocking.
SOLVER_FAILUREandDIVIDE_BY_ZEROare not — the model imports and then fails to solve —while
MATH_GENERATION_FAILUREis, and that is what both now report.BIOMD0000001065needed no change: it went
FAIL→PASS, andSOLVER_FAILUREwas already non-blocking, soits flag was
trueeither way.Regenerated with the command the test prints, so the file is byte-for-byte apart from the
two attributes, and stays CRLF.
What this does not fix — see #2077
Accepting a baseline does not stop these cases moving. Over the last seven nightlies four
different models deviated, in rotation:
1065FAILbaseline once, on 09-06547MATH_GENERATION_FAILUREon 09-04/05/06/07,SOLVER_FAILUREbefore613MATH_GENERATION_FAILUREon 09-03/05/06/07,DIVIDE_BY_ZEROon 09-04457PASSED→SOLVER_FAILUREon 09-05 only, and came backThe 08-30 pass and 08-31 failure are on the same commit, which rules out a code change.
Two are numerically borderline and probably always will be — 1065 fails on
pow(u,v) and u=-0.000000<0withEN_0_0 = -0.000000, and 547 onAt t = 8.09814e-18, mxstep steps taken.613 is not numerical, and that is what #2077 is about: it alternates between
divide by zero '(M_initConc / OC_initConc)'raised while writing the solver input andInitial condition for variable 'M' references variable 'L'raised during math generation— the same initial condition, two phases apart, same input and same code. That reads as
ordering-dependent math generation.
So expect this gate to fail again on a night when one of these lands the other way. That is
the gate working, not a new problem.
🤖 Generated with Claude Code
https://claude.ai/code/session_019HAnpFxkzf9LmxBayDSANf