Skip to content

test(bmdb): accept the three changed BMDB results - #2078

Merged
jcschaff merged 3 commits into
masterfrom
fix/bmdb-nightly-baseline
Sep 8, 2026
Merged

jcschaff merged 3 commits into
masterfrom
fix/bmdb-nightly-baseline

Conversation

@jcschaff

@jcschaff jcschaff commented Sep 8, 2026

Copy link
Copy Markdown
Member

The BMDB nightly has failed every night since 2026-08-31. This accepts the three deviations
in the 09-07 run (34094676265):

model baseline this run
BIOMD0000000547 SOLVER_FAILURE MATH_GENERATION_FAILURE
BIOMD0000000613 DIVIDE_BY_ZERO MATH_GENERATION_FAILURE
BIOMD0000001065 SOLVER_FAILURE PASSED

None is a regression. No model went from passing to failing; the two that changed were
already failing and now fail at an earlier phase, and one that was failing now passes.
Totals moved the right way: 914 → 915 passes, 147 → 146 failures.

Verified, both directions

Against that night's own exec_summary.ndjson rather than by reasoning about it:

  • old baselineFAIL: 3 changed, exit 1 — reproduces the nightly failure locally
  • new baselinePASS: no changes against the documented baseline, exit 0

Only the three rows are re-serialized; the other 1179 are untouched byte for byte, so the
diff is three lines.

A second file had to move with it

bioModelsNetInfo.xml carries a Supported flag per BioModels entry, and
BioModelsNetInfoTest keeps it honest against this baseline — it failed CI on the first
commit, which is the check doing its job. Two flags moved:

BIOMD0000000547  Supported true -> false
BIOMD0000000613  Supported true -> false

That is the right consequence, not a side effect. Supported means VCell can import the
model, so the test derives it from whether the recorded failure is import-blocking.
SOLVER_FAILURE and DIVIDE_BY_ZERO are not — the model imports and then fails to solve —
while MATH_GENERATION_FAILURE is, and that is what both now report. BIOMD0000001065
needed no change: it went FAILPASS, and SOLVER_FAILURE was already non-blocking, so
its flag was true either way.

Regenerated with the command the test prints, so the file is byte-for-byte apart from the
two attributes, and stays CRLF.

What this does not fix — see #2077

Accepting a baseline does not stop these cases moving. Over the last seven nightlies four
different models deviated, in rotation
:

model behaviour
1065 deviated (PASSED) on six of seven nights; matched its FAIL baseline once, on 09-06
547 MATH_GENERATION_FAILURE on 09-04/05/06/07, SOLVER_FAILURE before
613 MATH_GENERATION_FAILURE on 09-03/05/06/07, DIVIDE_BY_ZERO on 09-04
457 flipped PASSEDSOLVER_FAILURE on 09-05 only, and came back

The 08-30 pass and 08-31 failure are on the same commit, which rules out a code change.

Two are numerically borderline and probably always will be — 1065 fails on
pow(u,v) and u=-0.000000<0 with EN_0_0 = -0.000000, and 547 on
At t = 8.09814e-18, mxstep steps taken.

613 is not numerical, and that is what #2077 is about: it alternates between
divide by zero '(M_initConc / OC_initConc)' raised while writing the solver input and
Initial condition for variable 'M' references variable 'L' raised during math generation
— the same initial condition, two phases apart, same input and same code. That reads as
ordering-dependent math generation.

So expect this gate to fail again on a night when one of these lands the other way. That is
the gate working, not a new problem.

🤖 Generated with Claude Code

https://claude.ai/code/session_019HAnpFxkzf9LmxBayDSANf

The nightly has failed every night since 2026-08-31. These are the three
deviations in the 09-07 run:

  BIOMD0000000547  SOLVER_FAILURE  -> MATH_GENERATION_FAILURE
  BIOMD0000000613  DIVIDE_BY_ZERO  -> MATH_GENERATION_FAILURE
  BIOMD0000001065  SOLVER_FAILURE  -> PASSED

None of them is a regression: no model went from passing to failing, and the
totals moved the right way, 914 -> 915 passes and 147 -> 146 failures.

Verified both directions against that night's own exec_summary.ndjson rather
than reasoning about it - the old baseline reproduces the failure locally
("FAIL: 3 changed", exit 1) and the new one passes ("PASS: no changes against
the documented baseline", exit 0).

Only the three rows are re-serialized; the other 1179 are untouched byte for
byte, so the diff is 3 lines.

WHAT THIS DOES NOT FIX. These cases are non-deterministic, and accepting a
baseline does not make them stop moving. Over the last seven nightlies four
different models deviated, in rotation:

  1065  deviated (PASSED) on six of seven nights; matched its FAIL baseline
        once, on 09-06
  547   MATH_GENERATION_FAILURE on 09-04/05/06/07, SOLVER_FAILURE before that
  613   MATH_GENERATION_FAILURE on 09-03/05/06/07, DIVIDE_BY_ZERO on 09-04
  457   flipped PASSED -> SOLVER_FAILURE on 09-05 only, and came back

The recorded failures say why for two of them. 1065 fails on
"pow(u,v) and u=-0.000000<0" with EN_0_0 = -0.000000, and 547 on
"At t = 8.09814e-18, mxstep steps taken" - both are values sitting on the edge
of zero, where the sign of the last bit decides the outcome.

613 is the one worth looking at, because it is not numerical. It alternates
between "divide by zero '(M_initConc / OC_initConc)'" raised while writing the
solver input, and "Initial condition for variable 'M' references variable 'L'"
raised during math generation - the same initial condition, surfacing two
phases apart. That reads as ordering-dependent math generation rather than
arithmetic. Filed separately.

So expect this gate to fail again on a night when one of these lands the other
way; that is the gate working, not a new problem.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019HAnpFxkzf9LmxBayDSANf
jcschaff and others added 2 commits September 7, 2026 22:28
bioModelsNetInfo.xml carries a Supported flag per BioModels entry, and
BioModelsNetInfoTest keeps it honest against the nightly baseline. Accepting
the new failure types moved two of them:

  BIOMD0000000547  Supported true -> false
  BIOMD0000000613  Supported true -> false

That is the correct consequence rather than a side effect. Supported means
VCell can IMPORT the model, so the test derives it from whether the recorded
failure is import-blocking. SOLVER_FAILURE and DIVIDE_BY_ZERO are not - the
model imports and then does not solve - while MATH_GENERATION_FAILURE is, and
that is what both of these now report.

BIOMD0000001065 needed no change for the same reason: it moved FAIL -> PASS,
and SOLVER_FAILURE was already non-blocking, so its flag was true either way.

Regenerated with the command the test itself prints, so the file is rewritten
byte for byte apart from the two attributes; it is CRLF and stays that way.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019HAnpFxkzf9LmxBayDSANf
Comments only; no behaviour change.

There are two files here and they are easy to mistake for one, because they
overlap in vocabulary and both talk about which BioModels archives VCell can
handle. They answer different questions:

  test_cases.ndjson    the nightly's expected-results baseline. Source of
                       truth for what each case did last time we accepted it,
                       and so both a regression gate and a compliance record.

  bioModelsNetInfo.xml a prediction, shipped with the client and read at
                       runtime, of which archives the BMDB tab expects to
                       OPEN. Derived from the first; not a source of truth.

I conflated them while accepting a baseline change, argued the first one's
semantics at the second one's file, and had to be corrected. So the
definitions now live on the data models rather than in anyone's memory:
OmexTestCase, OmexExecSummary, FailureType, BioModelsNetModelInfo, the two
XML attribute constants, and a header comment in the XML itself.

Three things worth writing down that the code does not say anywhere:

- The XML predicts OPENING, not running. The tab uses the ordinary
  document-open path and never simulates, so ~70 archives that import cleanly
  and then fail in the solver are correctly still Supported="true". Reading
  the flag as "will run" makes those look like bugs.
- FailureType mixes three kinds of thing with only the naming to tell them
  apart: capability statements (UNSUPPORTED_*), defects, and operational
  states. Nothing enforces it, which is why BioModelsNetInfoTest carries its
  own explicit set. The enum comment now asks that new values put the kind in
  the name, since an UNSUPPORTED_ prefix reads as a promise about scope.
- Status.SKIP already means two different things - excluded because too slow,
  and not applicable because we do not support the feature - distinguishable
  only by the description text.

Also records the two dead attributes rather than leaving them to be
rediscovered: exception= is read by nothing, which is why the tab can say a
model is not compatible but not why; Slow= is read, but sits on three rows
that are the first three of the eleven the nightly excludes for the same
reason, never reconciled.

One trap found writing this: "--" is illegal inside an XML comment, and the
client parses this file with JDOM at runtime. BioModelsNetInfoTest did not
catch it, because it is line-based rather than an XML parse.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019HAnpFxkzf9LmxBayDSANf
@jcschaff
jcschaff merged commit 0654459 into master Sep 8, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant