perf(convert): decode the mzML in parallel, and stop calling a whole file truncated

[RobbinBouwmeester]

convert is strictly serial inside the per-run chain, so nothing overlaps it: 63.7 minutes of the seven-file immuno experiment, and 29 seconds per Astral file. It is now about ten times faster, and it no longer rejects a valid file.

The correctness bug, taken first

A non-finite retention time in the last spectrum made convert refuse the whole mzML as truncated or corrupt. The read counter was the last statement of the loop body and the non-finite arm reached it through continue, so the count came out one short of the header's declaration and the guard fired. The counter moves to the top of the body, after the --max-spectra break so the cap still evaluates identically. The test pins the old behaviour: it fails on the previous code with exactly that message. Siblings cover the mid-file case, a genuinely short file, and the cap.

The parallel decode

decode_one (per spectrum) is split from Fold::absorb (all order-dependent state, one thread, index order). The offset index MZReader::open_path already builds and then discarded now cuts the file into ~1 MiB chunks; worker w takes chunks w, w+workers, … on its own reader, and one bounded channel per worker lets the fold consume strictly in order with no reorder buffer.

Measured on LFQ_Orbitrap_AIF_Ecoli_02.mzML, 1.544 GB, 236,042 spectra, 32 cores, medians of 3-4 runs:

decode threads	stage elapsed
sequential	14,490 ms
1 worker	9,900 ms
2	3,319 ms
4	1,859 ms
8	1,414 ms
32	1,479 ms

A second file goes 12,701 to 1,544 ms. Note the one-worker column: about a third of the sequential time was mzdata's 10,000-byte read buffer, not the serialism, which the survey had folded into its estimate.

It is gated on the file agreeing with its own index: mzML, an index as long as the header count, and a two-spectrum probe of the last and middle spectra. A nonconforming file that reports index 0 for every spectrum falls back to sequential rather than failing, and there is a test for it. MUMDIA_CONVERT_THREADS overrides; 0 or 1 forces the old path. Peak working set is unchanged at about 9.5 MB, because chunks are sized in file bytes rather than in spectra.

Two smaller items

The zlib backend now really is zlib-rs. The obvious spelling selects the opposite codec: mzdata/zlib-rs also enables flate2/zlib, and flate2 ranks the C backend above zlib-rs. cargo tree on the shipped spelling shows zlib-rs 0.6.8 with no libz-sys and no new cc. Separately measured at -20% at two threads, -11% at four, and invisible at 32, where the stage hits a serial floor of about 1.4 seconds in the index read, the parquet encode and the hashing.

msconvert is asked for --mz64 --inten32 instead of --64, so intensities are not inflated to 64 bits and halved again. No measurement: msconvert is not on this machine and every workload here is supplied mzML or Thermo.

Equality

Byte-identical, asserted rather than argued. Every float in decode_one is per spectrum, so no reduction is reordered, and the counters fold in the same order. Three unit tests diff all four parquet artifacts and their content hashes between the two paths, including with the cap and with a wrong index attribute; both real 1.5 GB AIF files are identical sequential against parallel; and ci/smoke.sh gives the same peptides.tsv and proteins.tsv hashes with MUMDIA_CONVERT_THREADS=1 and with the default, on both its ungrouped and grouped arms.

cargo fmt --check, clippy -D warnings, 401 tests, cargo build --release --locked, ci/smoke.sh → SMOKE_OK, and the licence, SBOM, CLI-reference, doc-reference and workflow checks all pass.

🤖 Generated with Claude Code