fix(groups): fit the fragment mass calibration once on the pooled deviations - #113
Merged
Merged
Conversation
…iations
A grouped search learned a fragment tolerance 35% too wide, and it cost
identifications. Six-file HYE Astral, same library and the same retention-time
model, 100 isolation-window bands against none:
unbanded 100 bands
frag_ppm_offset -1.8486 -1.8834
frag_ppm_sigma 8.452 11.400
n_dev 181,196 200,257
ppm_residual_mad 0.907 1.035
candidates accepted 4,986,153 6,609,984
peptides at 1% 113,860 110,006
precursors at 1% 126,436 121,966
Causal rather than correlated: one band extracted twice, identical in every input
but the calibration file it was handed, accepted 12,414 candidates at 11.40 ppm and
8,791 at 8.45; an unbanded control on the same adapted library returned the unbanded
arm's 113,860 peptides exactly.
search-seed fits 1.5 * p95(|dev - median|) over the ppm deviations of the matched
fragments of its confident targets. Each band did that on its own ~2,000 deviations
and seed-pool combined the bands' SCALARS, which is not that estimator: a small
sample's p95 has a heavier tail than the union's, and each band selected calibrants
on its OWN q (106,088 band-confident against 97,584 pooled).
So fit once over the union, as the retention-time calibration already does with the
pooled anchors under groups.calibration = global:
- the estimator, the masscal.json body and the calibrant sidecar move into
masscal.rs, so the two call sites share one function rather than two copies of the
percentile logic;
- search-seed, when asked (SearchSeedParams::emit_calibrants, set only by the
grouped path), writes <seed>.masscal.parquet: library-wide candidate_id,
scan_index, frag_mz, ppm, one row per matched fragment, 16 B each;
- seed-pool keeps the deviations whose PSM the POOLED q accepts at fdr_seed, and
whose scan the pool kept, so an overlap candidate contributes one PSM's fragments
exactly as an ungrouped seed would, then fits MassCal::fit_from once.
two_pass_mass_cal and the mass_cal_loess grid keep working on this path because
both read the deviations rather than a scalar;
- a missing sidecar, which is a band directory from an older build, keeps today's
scalar combination and says so in the log, with a second warning when
mass_cal_loess cannot be honoured from scalars. masscal.json records which path
ran in masscal_source.
The band q is not the pooled q in either direction, and where it is STRICTER the
pooled filter alone would starve the fit: on the CI fixture at three bands every
band's own q rejects every one of its targets, so a strictly q-selected sidecar is
empty in all three. The sidecar therefore also offers the band's best-scoring
targets down to masscal::CALIBRANT_OFFER_PSMS (2,000) and lets the pooled q choose.
The three bands then contribute 437 + 401 + 230 = 1,068 deviations and the pool fits
frag_tol_ppm 5.0 at offset 0.0, which is to the digit what the ungrouped search of
the same spectra produces. Before this the same run calibrated nothing and extracted
at the configured 20 ppm.
What stays identical: an ungrouped run. emit_calibrants is false at all three
ungrouped call sites, no sidecar is written and the calibrant selection is
unchanged. Verified by running ci/smoke.sh from this tree and from a816768 and
comparing all 223 output files: every artifact byte-identical, only elapsed_ms,
git_sha and embedded work-directory paths differ.
What deliberately moves: a grouped run's mass calibration, and every artifact
downstream of it (extract, features, compete, rescore, quant, report). That is the
point of the change.
Validated from rust/mumdia: cargo fmt --check, cargo clippy --workspace
--all-targets -D warnings (clean), cargo test --workspace (319 lib tests), plus
ci/smoke.sh (SMOKE_OK) and a three-band grouped run on the fixture.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A grouped search learned a fragment tolerance 35% too wide, and it cost identifications. Measured on the six-file HYE Astral benchmark, same library and the same retention-time model, 100 isolation-window bands against none:
Why
seed_poolcombined the BANDS' fitted scalars by calibrant count. The tolerance is1.5 × p95(|dev − median|), which does not average: a 95th percentile over one band's ~2,000 deviations has a heavier tail than the same percentile over the union. Each band also selected its calibrants on its own q, whose 1/T floor is looser than the pooled one, so the bands counted 106,088 confident seed PSMs where the pool counts 97,584.Causal rather than correlated: one band extracted twice, identical in every input except which calibration file it was handed, accepted 12,414 candidates at 11.400 ppm and 8,791 at 8.452. A control arm, unbanded on the same adapted library, reproduced the unbanded arm's 113,860 peptides exactly, which rules out every other difference between the arms.
What changed
The estimator now exists once, in a new
masscalmodule that both the seed and the pool call. A band's seed writes its calibrant deviations beside its calibration: library-widecandidate_id,scan_index, fragment m/z and the ppm deviation, 16 bytes each.seed_poolreads them, keeps the deviations whose PSM the POOLED q accepts, and fits one offset and one tolerance. A band directory without a sidecar still pools on the scalars, warns, and records which path it took inmasscal_source.One thing the implementation found that the diagnosis missed. The band q is not the pooled q in either direction: on small bands it is far STRICTER, and on the CI fixture every band's own q rejects every target, so a strictly q-selected sidecar was empty in all three bands and the pooled fit got nothing, falling back to the same 20 ppm default the change exists to remove. The sidecar therefore also offers the band's best-scoring targets down to a floor of 2,000 PSMs, and the pooled q decides which of them count. On the fixture that gives 1,068 deviations and a calibration digit-for-digit identical to the ungrouped one for the same spectra. A band's own
masscal.jsonstill fits on its own confident targets, socalibration: per_groupis untouched.Equality
An ungrouped run is byte-identical:
ci/smoke.shrun from this tree and froma816768, all 223 output files hashed, zero content differences, and no calibrant sidecar is written anywhere in an ungrouped run. A grouped run's calibration deliberately moves, and every artifact downstream of it with it.Per CLAUDE.md this is a correctness fix, the previous value being wrong on its own terms, not a promoted sensitivity default: it still wants entrapment and a second acquisition before anyone quotes it as a sensitivity result. The banded arm is rerunning on this branch now to measure what comes back.
cargo fmt --check,clippy -D warnings, 319 + 80 tests,cargo build --release --locked,ci/smoke.sh→ SMOKE_OK, and the three generator checks all pass.🤖 Generated with Claude Code