Conversation
MsScanMatchResultHelper.AreEqual is the only field-by-field round-trip assertion for MsScanMatchResult, and it was three serialized members behind: CollisionEnergy, EnhancedDotProduct and SpectralEntropy carry [Key(36..38)] and were compared by nothing. Extending it was not enough. SaveAndLoadTest, the only test that exercises it, builds its fixture from `new MsScanMatchResult()` and sets Source and TotalScore alone, so the other assertions compared a default against a default and could not fail. The witness existed and watched nothing -- a member added with a duplicate or omitted [Key] would have round-tripped untested and the suite would have stayed green. So the fixture now carries a distinct non-default value for all thirty-eight serialized members, and a reflection check asserts that it does. That check is the part worth keeping: it fails when a new [Key] member is added above without being given a value, which is the moment the omission is cheap to fix rather than the moment it silently costs a round trip. This lands before the members it exists to protect, so the helper's own edit is proven separately from what comes next. 1466 tests pass across the ten suites, against 1465 before. No expected value was edited. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four MessagePack keys on MsScanMatchResult, written by nothing yet and read by nothing at all: MeasuredTerms says which similarity terms were computed, EvidenceSource says what kind of evidence the annotation rests on, and the two counts say how many candidates this annotator scored for the peak and how many passed its thresholds. Keys 39-42; key 25 is a hole that predates this repository and stays a hole. The evidence source is a new enum rather than a SourceType bit. SourceType is a [Flags] byte with two free bits for five evidence kinds, and it already answers several unrelated questions -- which database, whether a person intervened, whether anything matched -- for the ranking key, the exporters and the deserialization strip. It is also not IsReferenceMatched, which reads as "a spectrum matched" and is not: the text-database annotators set it on precursor m/z alone and a manual acceptance sets it on anything a person accepted. That boolean keeps its meaning and its value; this records the evidence beside it. The candidate count is the member that justifies the commit. The annotation processes keep only the best few candidates, so the size of the population is destroyed before any file exists, and without it "the run could not discriminate between candidates" cannot be told from "the run never reported more than a handful". Whether the list was truncated is left to the reader to derive, because the cap differs between the annotation processes and one flag would be wrong for most of them. The two counts are int? rather than int = -1, and finding out why was the useful part. MessagePack's generated deserializer assigns default(T) to every key absent from a short array -- it does not fall back to the C# property initializer, contrary to what one might reasonably assume from reading the type, where LibraryID = -1 and its siblings suggest otherwise. So -1 does not survive a project written before key 41 and 0 does, which would have every such project reporting that zero candidates were scored. A new backward-compatibility test caught it by re-encoding a genuine payload at the old length, and now pins both directions: an older project loads and asserts no evidence, and a payload from a future build is skipped past rather than throwing. The .dcl path is untouched and cannot be reached by this change: its record size comes from an explicit list of members, not from reflection over the type. 1469 tests pass across the ten suites, against 1466 before. No expected value was edited. The WPF GUI and the net48 Console both build with no errors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every place that computes a similarity term now says which terms it computed, at the point it computes them. No value changes and no branch is taken differently; nothing reads the record yet. The reason to record it at the measurement site rather than infer it afterwards is that the evidence is destroyed on the way out. The dot-product getters clamp the not-computed -1 to 0, so a spectrum that was never opened is indistinguishable from one that scored 0. Ms2MatchResult.Empty is worse: the calculator returns it precisely when the scoring functions gave -1, and it holds 0 in every spectral field, so Assign writes a plain 0 with nothing to mark it. Both now carry an explicit flag instead. Recorded in nine places -- MsReferenceScorer, MassAnnotator, the Imms/Lcimms/Lcms initializers, the four IMatchResult.Assign paths, the shared MsScanMatching leaf that every basic and EI comparison funnels through, and the target-formula path in IsotopeTracking. The DIMS annotators need no change of their own: they compose their results from the Assign paths. The composition rule lives once, in MeasuredTermsExtension, because these nine sites differ only in which terms their acquisition mode has and an omission in one leaves that whole mode's evidence blank. Two rules, not one, because MsScanMatching has two GetGaussianSimilarity overloads. The four-argument one returns the sentinel when either side is missing. The three-argument one does not: it evaluates the exponential against whatever it is given, so a reference with no retention time gets a similarity computed against 0 and the value keeps no trace of the absence. Most annotators call that one, so for them the flag is set from the inputs. That means the flag deliberately says "not measured" where the stored similarity is non-zero. The flag is the true statement; making the value agree moves scores, so it is a separate change. 41 witnesses across six test assemblies, one per measurement site, asserting facts and not values. Checked against three mutations of the recording rule -- never record, record unconditionally, and Assign records nothing -- each of which turns the relevant assemblies red. The first pass of that check found the mode-specific tests were exercising only the enabled/disabled guard, so each mode also got a case where the term is enabled and there is nothing on the reference side to compare. All ten suites green: 1510 passed, 2 skipped, up from 1469. WPF GUI and net48 Console build with no errors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
EvidenceSource is now written at every site that produces an annotation. No value changes and no branch is taken differently; nothing reads it yet. Sixteen sites, found by sweeping the repository from five independent angles rather than by grepping for the constructor. Two of them a constructor grep cannot find: CalculateMatchScore.CalculateMatches gets its result back from MsScanMatching and overwrites two members, and it is the single funnel for the whole GC-MS mode; DimsMspAnnotator needs its own assignment even though the previous commit's term recording reached it for free, because a calculator sees a query and a reference and cannot know what kind of database the reference came from. The lipid path records RuleBased, not ReferenceSpectrum. Its .lbm2 spectra are generated in silico, and the spectral comparison is a permissive pre-filter rather than the evidence -- ValidateBase combines the three dot products with OR for TargetOmics.Lipidomics where metabolomics uses AND, the default cut-offs are low, and ValidateOnLipidomics then requires a characteristic-ion rule to have fired. The rules in MsmsCharacterization establish both whether the match stands and at what structural level. Keyed on TargetOmics, which is already a field at every MSP site, so no DataBaseSource plumbing is needed -- which is as well, because MassAnnotationSettingModel hard-codes DataBaseSource.Lbm on a branch shared with plain MSP. The values describe what was compared, not what the database holds. An MSP candidate for a feature with no product-ion spectrum is PrecursorOnly. The alternative reading would let a row be labelled "a reference spectrum was compared" while its own name carried the "no MS2: " prefix that AnnotationName already applies to exactly that case, and it would contradict the programme's own policy wording, in which a reference match outranks a precursor-only suggestion. Which database a name came from is a separate question that Source already answers. Manual gets exactly one site: marking a peak unknown. A person accepting a candidate does NOT relabel it. The object they accept is the annotator's own instance, stored by reference with no clone anywhere in the GUI write path, so stamping Manual there would overwrite the only record of what was compared in exchange for a fact SourceType.Manual and IsManuallyModified already carry. Producer owns, acceptance does not. MS-FINDER's reflect-to-MSDIAL records PredictedSpectrum. Today an in-silico fragmenter result enters the molecule database and reaches the container with IsReferenceMatched true, indistinguishable from an experimental match; Source there is bare Manual, so the human step was recorded twice and the in-silico step nowhere. That site also filled RtSimilarity and RiSimilarity without recording the terms, which the previous commit missed; recorded here. Two blanks are deliberate and reasoned in the code: MsScanMatching records nothing, because its leaf serves metabolomics, lipidomics, EI and proteomics alike and CompareEIMSScanProperties is also how GcmsPeakJoiner compares two sample spectra during alignment, where there is no reference at all; and the .dcl reader and writer record nothing, because that block has a fixed byte layout. Known limitation, pinned rather than hidden: the .dcl annotation block carries only numeric scores, IDs and booleans -- no strings, not even Name -- so the GC-MS file process, which saves and reloads through it, discards the evidence record. ADclRoundTripDiscardsTheEvidenceSource asserts that loss so it is a named fact. 30 further witnesses, one per site per mode, including the first test anywhere to reach a GeneratedLipid scorer. Checked against two mutations of the recording rule -- ignore the omics, and drop the spectrum guard -- each of which turns all five mode projects red. All ten suites green: 1615 passed, 2 skipped, up from 1510. WPF GUI and net48 Console build with no errors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The candidate counts are now written at every site that stores an annotation from a searched population. No value changes and no branch is taken differently; nothing reads them yet. This is the one fact in the evidence record that cannot be recovered from anything else. NUMBER_OF_ANNOTATION_RESULTS is 3, the IMMS and proteo-metabolomics processes keep one, GC-MS keeps five -- so the size of the candidate population is destroyed before any file exists, and "the run could not discriminate between forty isomers" has been indistinguishable from "the run reports at most three". Three tiers, not two, on the developer's own suggestion: filtering the threshold-passing list by IsReferenceMatched is exactly what SelectReferenceMatchResults computes, it is free, and neither figure survives truncation. So key 43 joins 41 and 42, and the differences give what the annotation policy distinguishes -- rejected outright, precursor -only suggestions, MS/MS reference matches. A related finding is written into key 42's remarks: FilterByThreshold applies no threshold of its own. It is IsAnnotationSuggested || IsReferenceMatched, and MsScanMatchResultEvaluator's constructor ignores its search parameter entirely, so the cut-offs were applied inside the annotator. Nine sites across six processes. CandidatePopulation carries the rule because more than one population is live at once in three of them: the EAD lipid process runs a molecular-species query and then a generated-lipid query per factory, and giving one population's numbers to the other's results would be a false record nothing downstream could detect. Constructing it as a value forces the author to name which one they mean, and it takes the run's own IsReferenceMatched predicate rather than reading the property, so the count cannot disagree with the selection the caller goes on to make. No .ToList() was needed after all: IMatchResultFinder.FindCandidates is declared to return List<TResult>, so Count() is O(1) everywhere. That matters more than economy at one site -- EadLipidAnnotator.FindCandidates reaches EadLipidDatabase.Generates, which assigns ScanIDs and registers references, so a second call to count would change existing values, not merely double the work. The comment there says so. GC-MS is the one exception and does materialise, because CalculateMatches yields. Two contract corrections after review, both to statements that would have misled the first consumer: - The counts are per (peak, annotator, product-ion spectrum), NOT per (peak, annotator) as key 41 previously claimed. AIF acquisition runs the same annotator once per collision energy, and LC-IM-MS once per drift peak, so one container legitimately holds several populations from one annotator. Consumers must group by (AnnotatorID, SpectrumID) and take a distinct value, never a sum. - An annotator that scored candidates and named none of them stores no result, so its population has no carrier and is unrecordable here, not merely unrecorded. Any total over visible rows is a lower bound of unknown looseness. Stated on key 41 and on the test that pins the arithmetic for that case. Also corrects this branch's own previous commit, which overstated the .dcl limitation: GC-MS does keep the evidence record. The primary store is MessagePack through AnnotatedMSDecResult's formatter, and that is the copy GcmsAnalysisMetadataAccessor exports and DataObjConverter merges into alignment. The .dcl block is a second, lossy copy -- it carries no strings at all, not even Name -- reached only through the legacy MSRawID2MspBasedMatchResult dictionary. Deliberately blank, each with the reason in the code: the GUI compound-search dialogs (no threshold runs, and the MS-scan dialog scores the whole library rather than a window, so a count there would be a third meaning for one field); the target-formula matcher in IsotopeTracking (the search runs the other way round, windowing spots around a query); the peptide path, on the developer's instruction that proteomics annotation criteria differ categorically; and the dead LC-IM-MS process, which is also the only place candidates.Count would be a post-filter number. 9 witnesses, checked against three mutations -- record nothing, read the property instead of the predicate, and confuse the population with the named subset -- each of which turns both levels red. The first pass of that check found the process-level test could not catch the third, because the mock's FilterByThreshold passed everything through and made the two numbers equal; it now filters as the real evaluator does. All ten suites green: 1547 passed, 2 skipped. WPF GUI and net48 Console build with no errors. Correcting the record: the previous commit's message says 1615 passed and 30 witnesses. Both were arithmetic errors of mine. That run was 1538 passed with 28 new witnesses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
No production code in this commit. It records what MS-DIAL does today at the four places phase 1b will change, so that each change is a visible, deliberate edit to a named assertion rather than a silent movement in output. Every test passes now and is expected to fail when its fix lands; each names the correct value on the assertion, and the set is findable with --filter TestCategory=PinnedDefect. The retention-time term, which the author has confirmed was never intended. A reference with no retention time carries ChromX.RetentionTime.Default, whose value is -1, and the unguarded three-argument GetGaussianSimilarity evaluates the exponential against it as though it were a measurement. Two findings beyond what was expected. The fix has TWO sites, not one: IsRtMatch is computed separately by a Math.Abs comparison in ValidateBase, so guarding the Gaussian alone would leave the verdict wrongly true -- and it is true here. And the damage runs the opposite way from the obvious guess: the score is an average over the included terms with the MS/MS term weighted 3, so a fabricated retention-time term DILUTES it. 1.939 becomes 1.613. A reference with no retention time is penalised, by an amount that depends on how early the peak eluted. With a five-minute tolerance the fabricated similarity is 0.96. The text-database verdict. A text-database hit is marked IsReferenceMatched on precursor-mass agreement alone, and the evidence record added in the preceding commits sits on the same object saying EvidenceSource is PrecursorOnly and no spectrum was compared. The contradiction is pinned as a single assertion, so what fails when the flag is fixed is the contradiction itself. LC-MS is asserted because it is the campaign's mode; the other three text-database annotators behave the same way and each already has an evidence witness in its own project, so a fix must visit all four. The export divergence, which is live and was not what the plan described. A genuine total score of exactly zero is written as "0.000" by the analysis file and as "null" by the alignment file. The cause is overload resolution, which is why centralising the spectral columns in AnnotationScoreFormat did not catch it: IMetadataAccessor's ValueOrNull(float) and ValueOrNull(double) both return "null" below 1e-10, while ValueOrNull(double?) tests only for null -- and the two accessors differ by one question mark, matchResult?.TotalScore against matchResult.TotalScore. The analysis file is right by this codebase's own rule, since TotalScore is always computed. Reachable in production: scores.DefaultIfEmpty().Average() is exactly 0 when no term was included. SharedScoreColumns in that test class already omits "Total score", so the existing agreement test does not cover it. The truncation cap, which is characterization rather than a defect. Four threshold-passing candidates, a cap of three, and the lowest-scoring reference match is discarded leaving no other trace -- so this also demonstrates what the candidate counts recover: each stored row reports four above threshold while three rows exist. It will fail if the cap or the ordering changes, which is the point. One correction to my own work here: the text-database contradiction test was first written to fail today, which would have left the suite red and hidden real regressions. Inverted, with the post-fix form in the comment. All ten suites green: 1560 passed, 2 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A MoleculeMsReference with no retention time holds ChromX.RetentionTime.Default, whose value is -1. Two places treated that -1 as a measurement: the unguarded three-argument GetGaussianSimilarity, which evaluated exp(-0.5*((rt - -1)/tol)^2) on it, and a separate Math.Abs comparison in each annotator's ValidateBase, which set IsRtMatch on the same grounds. An early-eluting peak therefore agreed, on retention time, with a reference that had none -- and agreed more strongly the earlier it eluted. Confirmed as unintended by the author, 2026-09-10. The damage was not mainly the total score. DataAccess.GetAnnotationCode reads IsRtMatch and nothing else to promote 430 (m/z + MS/MS matched) to 330 (RT + MS/MS matched), and that code is written to the "Annotation tag (VS1.0)" column of every analysis and alignment export. The fabricated agreement published a claim of retention-time confirmation for references that had no retention time to confirm. The score movement was a side effect by comparison, and it went the unintuitive way: MsReferenceScorer averages the included terms and the MS/MS term carries a factor of 3, so the extra term diluted the average and LOWERED the score, penalising such references by an amount that depended on how early the peak eluted. The rule is an exemption, not a rejection, and RetentionMatchPolicy states it once. Simply forcing IsRtMatch false would have deleted these candidates rather than demoting them: IsReferenceMatched and IsAnnotationSuggested share the retention clause, FilterByThreshold is their disjunction, and StandardAnnotationProcess stores only what that returns. In a library where only some entries carry retention times -- the case this was reported for -- every entry that does not would have vanished from the output, taking its m/z and MS/MS evidence with it. MeasuredTerms.RetentionTime, added in the preceding commits, is what supplies the third state a boolean cannot hold: set with IsRtMatch true means agreed, set with false means disagreed, clear means no comparison was possible. Only the middle case may withhold a verdict. MassRtReferenceSearcher gets the same exemption on the filtering side, where the behaviour used to flip on the tolerance: a narrow window silently dropped every reference without a retention time, while the 100-minute default admitted them all and handed them to the unguarded Gaussian. The mass range has already bounded that set, so exempting them costs nothing. Sites: MsReferenceScorer (LC-MS MSP, lipidomics, EAD), LcmsTextDBAnnotator, LcimmsMspAnnotator, LcimmsTextDBAnnotator, and the two verdicts re-derived after scoring in LcmsMspAnnotator and EadLipidAnnotator. Not touched. GC-MS needs no change: CompareBasicMSScanProperties, the single funnel for every EI comparison, already calls the guarded four-argument overload for both retention time and retention index. Collision cross section has the same shape and is deliberately left alone. LcmsFastaAnnotator is left alone because proteomics annotation criteria are separate by decision. FabricatedRetentionTimeTermTests pinned the old behaviour and named the correct value on every assertion; all four of its defect assertions failed on this change, with the predicted values (RtSimilarity -1, IsRtMatch false, and the two totals converging on 1.93876). It is replaced by MissingReferenceRetentionTimeTests, which asserts those values plus the confidence code staying at 430, the candidate surviving FilterByThreshold, and -- as the guard against an over-broad exemption -- a reference whose retention time genuinely disagrees still being rejected. 12 suites: 1606 passed, 0 failed, 5 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A search tolerance is not a match criterion. RtTolerance says which references
are worth scoring and its default is 100 minutes; the verdict says whether the
retention times actually agree. Ticking "use retention time for scoring" while
leaving that default meant every reference in the library agreed on retention
time, which promoted the exported confidence code from 430 to 330 (and 440 to
340 on the GC-MS branch, where DatasetParameterSettingModel turns retention
scoring on by default). Searching a predicted-retention-time library loosely is
reasonable; publishing "retention time matched" on the strength of that search is
not.
So the verdict is capped and the score is not. The Gaussian keeps the user's
tolerance as its width, which is what stops this from silently re-ranking
existing projects: TheGaussianStillUsesTheUsersTolerance pins RtSimilarity at
0.99501 for a 10-minute difference under a 100-minute tolerance, the value it had
before this change and 5 orders of magnitude away from what a capped width would
give. Total scores do not move; verdicts and confidence codes do, which is the
intent.
Caps, chosen by the author, 2026-09-10:
retention time 2 minutes
Kovats (alkane) 150 index units -- one carbon is 100 units and about 1.1-1.3
minutes, so this is the same 2 minutes
Fiehn (FAME) 50,000 index units -- that scale is FAME retention in
milliseconds, near 39,350 units per carbon, so it is the same
order as 150 Kovats units
The two index scales differ by a factor of about 390, so RiCompoundType is a
required constructor argument on CalculateMatchScore rather than a defaulted one.
That choice immediately earned itself: the compiler found a fifth construction
site in the Console app that a default would have left silently on the wrong
scale.
Sites: the four LC verdicts now call RetentionMatchPolicy.IsRetentionTimeMatch,
which also folds in the preceding commit's guard. GC-MS takes the cap through a
new optional argument on CompareEIMSScanProperties, applied before that function
derives IsReferenceMatched, so nothing has to be recomputed afterwards; the
default of MaxValue leaves every other caller alone, notably the peak joiners,
which compare two acquired spectra rather than annotating.
A consequence worth stating. Unlike a reference with no retention time, which is
exempt, a reference that has one and disagrees is rejected -- and since
IsReferenceMatched and IsAnnotationSuggested share that clause, rejected outright
rather than demoted to a suggestion. That is not new behaviour; it is what
already happened to anyone who set a realistic tolerance. What is new is that the
default no longer exempts everyone from it.
Not addressed: RiTolerance defaults to 20, which is a sensible Kovats window and
20 milliseconds on the Fiehn scale. Same unit ambiguity, separate defect, noted
in the test remarks.
Mutation-checked. Widening the retention-time cap to 1000 fails 3 tests; making
the Fiehn cap equal the Kovats one fails 2, including the behavioural one; and
dropping the cap argument at the CalculateMatchScore call site fails 2, so the
wiring is held and not only the constants.
12 suites: 1618 passed, 0 failed, 5 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Use retention time for scoring" and "use retention time for filtering" mean different things: scoring ranks candidates, filtering rejects them. The code had the scoring setting rejecting. IsReferenceMatched and IsAnnotationSuggested shared the retention clause, MsScanMatchResultEvaluator.FilterByThreshold is their disjunction, and StandardAnnotationProcess stores only what that returns -- so a candidate whose retention time disagreed was not demoted, it was deleted, with nothing left in the file for a reader to judge and no trace that a reference of that mass had been considered. IsAnnotationSuggested no longer carries the retention clause. That is the shape MassAnnotator and DimsMspAnnotator always had, so this makes the retention-using annotators consistent with the ones that do not use it. A retention-time disagreement now costs the reference match and leaves the precursor-only suggestion standing, which is what the evidence supports. This matters most for the preceding commit's cap. Capping the 100-minute default to 2 minutes, on top of the old shared clause, would have emptied the annotation output of every project that had ticked the box. It now lowers those rows instead. LcmsTextDBAnnotator and LcimmsTextDBAnnotator had no suggestion path at all -- ValidateCore set IsReferenceMatched and left IsAnnotationSuggested at its default -- so for them a disagreement was always fatal. They get one. That is the path a retention-time-anchored text database is used for in the public-repository campaign, and a text-database row is precursor-only evidence by construction, so a suggestion is exactly what it is. This is unrelated to the separately deferred question of whether such a row should be called a reference match at all (TextDbVerdictWithoutASpectrumTests), and does not touch it: with retention scoring off, every mass match is still a reference match and no new suggestion appears. Collision cross section is left alone deliberately, keeping its clause on both verdicts. ImmsMspAnnotator is the precedent -- the mobility axis gates suggestions there too -- and only the retention-time behaviour was asked to change. LcmsFastaAnnotator is untouched: proteomics criteria are separate by decision. Also corrects three remarks blocks that described the old shared-clause behaviour, including a <see cref> left dangling by a test rename in this series. Mutation-checked. Putting the clause back on MsReferenceScorer's suggestion fails 3 tests; removing the text-database suggestion path fails 2, including the one that asserts FilterByThreshold keeps the row. 12 suites: 1624 passed, 0 failed, 5 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Keys 39-43 have been recorded and serialized since the preceding commits but no export column existed for any of them, so the evidence record reached no file. This adds five columns -- Measured terms, Evidence source, Candidates found, Candidates above threshold, Candidates reference matched -- to both text formats in every mode except proteomics. Headers and content keys go in together and cannot be split. A header with no dictionary key throws KeyNotFoundException the moment an exporter indexes the content by header name; a key with no header is silently dropped. The two live in different methods, and in this tree they inherit differently: `base.GetContentCore` has ten callers, while `base.GetHeadersCore` has none at all -- every concrete accessor replaces the header array. So the content is added once in each base and the header list is edited in all thirteen places. Every column is supplied by an accessor. No exporter and no decorator learns the new names, and that is what keeps proteomics out of it: ProteomicsBaseAccessor implements IMetadataAccessor directly rather than deriving from the base, and it is handed to the same AlignmentCSVExporter as everyone else, so a column appended inside an exporter would demand a key it cannot supply and throw on every proteomics alignment export. Appending in the accessors avoids that entirely. Position is append-at-the-end, which is forced rather than chosen: LcmsMethodModel inserts "Ion mode" at a hard-coded index 34 with no name check, and appending is the only placement that leaves 34 meaning "MS/MS spectrum". AnnotationEvidenceFormat is the one place the rendering is decided, beside AnnotationScoreFormat and using the same "null" token. Three traps it exists to avoid, each with a test: MeasuredTerms.None and AnnotationEvidenceSource.Unspecified stringify to confident-looking words and must not be published as facts about runs that merely predate key 39; a count of 0 is a measurement and only null is not-recorded, which an accidental trip through ValueOrNull would destroy because two of its four overloads return "null" for anything within 1e-10 of zero; and Enum.ToString() on a multi-bit [Flags] value yields a comma, which AlignmentLongCSVExporter writes unquoted, so a CSV long export would gain phantom columns. Flags are joined with '|'. THE PARITY TEST IS THE POINT OF THIS COMMIT AS MUCH AS THE COLUMNS. Updating ten pinned expected-header arrays turns the suite green whether or not a value ever reaches a file, so each accessor test now also asserts that every header has a key and every key has a header, with an explicit declaredDrops list that must carry a reason. It does not forbid dropping a key; it forbids dropping one by accident. It found a real defect on its first run. "Enhanced dot product" and "Spectrum entropy" are computed for every analysis row by BaseAnalysisMetadataAccessor and listed in its own header array, but all four mode accessors replace that array and omit them -- so they have been computed and thrown away on every .mdpeak export in LC-MS, LC-IM-MS, DIMS and IMMS for as long as they have existed. Declared, not fixed: adding a column changes four modes' published column sets, which is a compatibility decision that does not belong here. The GC-MS alignment accessor drops ten more, all with reasons; those are deliberate. The two GC-MS accessors had no test at all and now do. GcmsAnalysisMetadataAccessor is the only accessor whose header array and content dictionary are both hand-written and local, so it is where a mismatch was most likely to reach a user. Not in this commit: mzTab-M, which reads six fixed keys and declares its own column lists, so opt_ columns there are a separate interchange decision; the per-candidate .mdcandidate.tsv sidecar, which uses the opposite absent-value convention -- the formatter is split into decision and rendering so it can reuse the decision; ResultExport, whose header and value lists are coupled positionally with no guard and which is dead code; and any reconciliation between MeasuredTerms and the existing sentinel-inference predicates. On that last point, worth knowing: the record is reported ALONGSIDE the inference, not made its source of truth, because switching the score columns to read the flag would move already-published cells and would be wrong for every project saved before key 39. So a row can show a blank rt_similarity, because the Gaussian underflowed, while Measured terms lists RetentionTime. That is one-directional and it is the flag being right. Mutation-checked: removing one content key from the base fails 4 tests naming the column, removing one header from one mode accessor fails 2, and making a zero count render as not-recorded fails 4 across the formatter, both bases and the cross-format comparison. 12 suites green: 1648 passed, 0 failed, 5 skipped. That run was made in a working tree that also carried an unrelated concurrent change, worth 7 of those tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Below threshold" was one bucket. At a 70% cut-off a candidate scoring 69% and a candidate that matched two fragments out of forty were recorded identically, and that is what makes the cut-off feel arbitrary: the cliff sat between "named" and "nothing" rather than between two grades of evidence. AnnotationEvidenceSource gains WeakSpectrumMatch and UnmatchedSpectrum, appended so stored values do not move, and a compared reference spectrum now lands in one of three tiers. NO NEW THRESHOLD IS INTRODUCED, deliberately -- the retention-time work in this series was caused by exactly one unexamined constant. For metabolomics the line between "fell short" and "explained nothing" is the analyst's own MinimumSpectrumMatch, the number their acceptance criteria already use, with a floor of one fragment for the case where they have set it to zero: a comparison that matched no fragment at all explained nothing by any reading. FOR LIPIDOMICS THE PEAK COUNT IS NOT CONSULTED AT ALL. A single diagnostic fragment can settle a lipid class outright -- cholesteryl ester is the example the author gave -- so a count near zero is entirely compatible with a correct class assignment, and asking the count there would demote a correct annotation to "explained nothing". The characteristic-ion rules are asked instead, which is consistent with those rules being the evidence on that path rather than the permissive dot-product pre-filter in front of them. WHY THIS IS A SECOND, LATER CALL rather than a change to the existing assignment. The verdict it reads -- IsSpectrumMatch and the lipid rule flags -- is not set until validation, and in MassAnnotator, ImmsMspAnnotator, LcimmsMspAnnotator and MsReferenceScorer validation runs in the CALLER of the method that produces the result, so grading at the production site would read flags that are not there yet. Moving six assignments and hoping none was missed fails unsafely: a missed site would leave the record blank. RecordSpectrumVerdict only ever downgrades an already-recorded value, so a site that never calls it keeps exactly the value it had before this commit -- wrong in the old way, never blank. Missing a site costs precision, not the record, and SpectrumVerdictTierTests holds the wiring for the path that matters most. The contract of key 40 widens from "what was compared" to "what was compared and what it supports", and nothing is lost by it: the narrower question is answered exactly by MeasuredTerms.Spectrum on the same record. Read as a pair the two keys say what was measured and what the measurement was worth. WHAT THIS IS FOR, in the author's words, 2026-09-10. A precursor-mass match with NO product-ion spectrum acquired can legitimately be reported -- at class level for a lipid, since chains cannot be resolved without MS2. A precursor-mass match WITH a spectrum that was acquired and failed is, by his criteria, unknown unless retention time supports it. Those are opposite verdicts and the suggestion machinery had been treating them as one bucket, both wearing a "no MS2:" or "low score:" prefix and both landing in IsAnnotationSuggested. The evidence record now separates them, so a consumer can apply either criterion; the displayed name is left alone, which was the author's choice, so no existing project loses a name. The Evidence source column added in 9a8a45d carries the new tiers to file with no further change. Not in this commit: the GC-MS EI path, which reaches evidence through WhenSpectrumCompared and has its own acceptance criteria; the proteomics path, separate by decision; and the mzTab-M writer, which is a published-format change and is being put to the author separately. Mutation-checked: dropping the wiring call in MsReferenceScorer fails 3 of the 5 integration tests, and letting lipidomics fall through to the peak count fails 3 unit tests including the cholesteryl-ester case by name. 12 suites green: 1664 passed, 0 failed, 5 skipped, in a tree that also carries an unrelated concurrent change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mzTab-M's chemical_name held whatever MS-DIAL had put in the feature name,
including the suggestion prefixes "no MS2: " and "low score: ". It now holds a
compound name, and the status goes to opt_global_evidence_source and
opt_global_measured_terms, where a reader can act on it instead of parsing it out
of a name -- and where it says more than the prefix could, since the prefix
collapsed "compared and fell short" and "compared and explained nothing" into one
word.
A CORRECTION TO WHAT I REPORTED EARLIER, and the reason this is not the change I
described. I said the pipe in a dual lipid name is manufactured in the
not-matched branch, so taking the last segment silently drops the "no MS2"
warning for lipids. That is true of ONE of the five sites that write a pipe.
MsScanMatching.GetRefinedLipidAnnotationLevel writes "class|chains" only at
annotation level 2 or above -- only when the characteristic-ion rules RESOLVED
the chains -- and returns the bare class name at level 1. MsReferenceScorer
writes the identical punctuation inside its if (!result.IsSpectrumMatch) branch,
where the chain half is copied off the reference and nothing measured it. Same
string, opposite evidence, so no rule phrased on the text can be right for both,
and the blanket "take the first segment" I was heading for would have demoted
every correctly chain-resolved lipid.
IsLipidChainsMatch can tell them apart, and it is set from the same call that
produces the name (MsScanMatching.cs:463-471), so the two cannot disagree.
AnnotationName.AtSupportedLevel takes the chains when the chains were resolved
and the class level when they were not. That is the author's rule of 2026-09-10 --
a lipid identified on precursor mass alone is "PC 34:1", not "PC 16:0_18:1" --
implemented from the record rather than from punctuation, and it leaves matched
lipids untouched.
The evidence gate on the SME section stops re-implementing the suggestion
vocabulary as a substring. It tested !Name.Contains("no MS2"), which caught one of
the three spellings: a "low score" spot -- spectrum acquired, compared, below the
criteria -- passed every gate and received a full rank-1 evidence row with
ms_level 2 and a complete score set, with nothing in the row saying it had failed,
and the peptide path's "w/o MS2" passed too. The gate now asks the evidence
record: a weak match keeps its row, because a comparison that partly agreed is
what the evidence section is for; a comparison that explained nothing, and one
that never happened, have nothing to cite. Projects written before the record
existed keep their old answer through the Unspecified arm, with the one
information-free correction that the test is a prefix rather than a substring and
covers "w/o MS2".
The normalisation and the vocabulary live in AnnotationName, which exists for
exactly this reason -- its own remarks say the writers and the reader are kept
together so a renamed prefix cannot leave a reader stale, and the pipe is part of
the same vocabulary.
Two opt columns, not five. The candidate counts describe how many alternatives
there were, and mzTab-M already says that in the ranked SME rows.
THE GOLDEN FIXTURE WAS NOT ENOUGH, and the mutation testing is what showed it.
Reverting the name normalisation left the byte-compared golden entirely green,
because the checked-in project contains no prefixed name, no dual lipid name and
no evidence record. The golden is regenerated here -- the diff is exactly the two
new columns and their values, all "null", which is itself the backward-compatible
arm working on a real pre-record project -- and five behavioural tests now cover
what it cannot. With those, reverting the normalisation fails 2 and restoring the
substring gate fails 1.
12 suites green: 1676 passed, 0 failed, 5 skipped, in a tree that also carries an
unrelated concurrent change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PredictedSpectrum described structure-to-spectrum prediction and was set by neither producer that used it. MS-FINDER, its only producer of consequence, runs the other way: measured peaks go in and a candidate structure comes out. CFM-ID and FIORA are the tools the old name actually described, and MS-DIAL has no annotator for them. The name was backwards. It becomes two members. ByStructurePredictionTool keeps the serialized value 3 -- renamed, not renumbered -- and is what MS-FINDER sets. BySpectrumPredictionTool is new at 8 and is what the peptide path sets, because PeptideMsReference.Spectrum is a b/y ladder computed from a sequence by SequenceToSpec: a structure went in and a spectrum came out, which is the other direction. They are kept apart because they fail differently. A spectrum predictor can be wrong about how a compound that is genuinely present fragments; a structure predictor can propose a compound that was never there. Those are not the same risk and an annotation record that merges them cannot express either. OUTSIDE MS-DIAL THEY ARE ONE TAG. The evidence inventory this programme publishes against records in-silico assignment as a single category, so the exported Evidence source column says "InSilico" for both, and which tool it was is already on the record beside it in AnnotatorID. Two members inside, one tag outside, decided by the author of MS-DIAL on 2026-09-14. No stored value moves: 3 keeps its meaning for the producer that set it, and 8 is appended. The only path whose stored value changes is the proteomics one, which moves from 3 to 8 -- correcting a misclassification rather than migrating data, and no project on this branch exists outside this work. Not addressed: where the two sit in candidate ordering. That is the next commit, and the author has settled it -- below WeakSpectrumMatch, since a comparison against real measured peaks outranks one against a generated spectrum. 12 suites green: 1679 passed, 0 failed, 5 skipped, in a tree that also carries an unrelated concurrent change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Candidate priority was quantitative all the way down: three orderings, and
below the verdict flags each of them fell through to TotalScore. So a name
suggested by precursor mass alone could beat a name a spectrum decided, on
the strength of a larger number.
The numbers make that likely rather than rare. A precursor-only result is
scored on fewer terms, and the terms it keeps -- accurate mass, sometimes
retention time -- are the ones that agree most easily, so their average
comes out above a genuine spectral comparison more often than not.
AnnotationEvidence.Rank puts the qualitative judgement in the ordering
instead, as the author asked on 2026-09-15: an order, not a weight. A
weight would make it tradeable, so a large enough similarity could buy a
precursor-only candidate past a spectral match; a separate key cannot be
bought at any score. The programme's annotation policy already says this
in words -- "a lower-priority MS/MS reference match outranks a
higher-priority precursor-only suggestion" -- and this is that sentence
compiled.
compared, and it carried the match 6 ReferenceSpectrum, RuleBased
compared, and it fell short 4 WeakSpectrumMatch
not recorded / asserted by a person 3 Unspecified, Manual
computed, or mass alone 2 the two in-silico members,
PrecursorOnly
compared, and it explained nothing 1 UnmatchedSpectrum
Three placements are the whole design.
"Not recorded" is mid-rank, not last. Every candidate deserialized from a
project written before the evidence record reads Unspecified -- absent
MessagePack keys give default(T) -- so in such a project every candidate
ties and the order is exactly today's, Argmax keeping the first of a tie.
Ranked last instead, reopening one and annotating into it would promote a
spectrum that explained nothing over a perfectly good older candidate,
purely because the older one predates the bookkeeping.
In-silico sits below WeakSpectrumMatch, confirmed by the author on
2026-09-14: a computed spectrum or a computed structure is a hypothesis
about a compound, a real spectrum that partly agreed is an observation of
one. It ties with PrecursorOnly rather than being placed above or below,
because whether a calculation outranks a bare mass has not been decided
and an ordering should not invent an answer.
UnmatchedSpectrum is below PrecursorOnly, which is the author's criterion
of 2026-09-10. No spectrum acquired is an absence of evidence; a spectrum
acquired, compared, and disagreeing is a finding against the candidate.
Rank is a switch and never a cast. The serialized values are a
compatibility record -- 3 is where PredictedSpectrum sat, 8 was appended
after it -- and run in the order the members happened to be written. Two
pairs are inverted between the two orders, so the test that a cast has to
fail is not hypothetical.
Where the rank goes, in each of the three keys:
MsScanMatchResultEvaluator below the verdicts, above the score. Below
them because they are the stronger statement, and because moving it
above would change what a text-database precursor-only match outranks --
a separate decision, deliberately untouched.
FacadeMatchResultEvaluator above Priority. Priority is the analyst's
ordering of their databases; IdentifySettingModel hands out
AnnotatorModels.Count - index, so the first library in the list won every
tie beneath Source, even against a later library that had matched a
spectrum where it had not. Source stays first, so a candidate a person
accepted still dominates: Manual is the top bit of the flags byte.
MsScanMatchResultContainer.ResultOrder same position. Without it the
change would be half-applied in the worst way -- SelectTopN would keep
the better evidence and the exported name would still come from the
higher-scoring candidate.
SelectTopN is not touched, as agreed. It holds no ordering expression and
defers entirely to SelectTopHit, which is why the cap now discards the
weakest evidence rather than the lowest score with no line changing there.
ResultOrder is also recomputed on load while the name it produced was
stored -- Representative is [IgnoreMember], AlignmentSpotProperty.Name is
Key(12). Reopening a project saved from this branch can therefore pair a
stored name with a different candidate's numbers. The author accepted that
on 2026-09-14 on the grounds that no such project exists outside this
development; reanalysis, not migration, is the remedy if one turns up.
Six mutations checked, each killed by tests that name the behaviour:
dropping the rank from any one of the three keys, ranking Priority above
it again, casting the enum, giving the DefaultIfEmpty placeholder the
neutral rank, and ranking "not recorded" last.
1,750 passed, 0 failed, 17 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DimsMspAnnotator averaged the mass term together with two spectral terms, always three of them. A feature with no product-ion spectrum -- the ordinary case in direct infusion, where an MS1 survey may be all there is -- got its two spectral terms from Ms2MatchResult.Empty, which holds 0 in every field rather than the -1 the scoring functions returned. So a precursor mass agreeing to within tolerance, worth 1.0 on its own, was published as 0.33. Below any TotalScoreCutoff above a third, which is most of them, so the candidate was not merely ranked low: it was discarded before anything could rank it. This is the only annotator that ever counted those zeros. MassAnnotator and MsReferenceScorer each assemble the terms they actually computed and average those, and so does CalculateAnnotatedScoreCore twenty lines below in this same file -- the same candidate, scored two ways, by two methods that disagreed. The fix is not a new convention; it is this site joining the existing one. The guard is Ms2MatchResult.SpectrumCompared, not IsSpectrumComparisonPerformed. That predicate infers "a comparison happened" from the spectral fields being non-negative, and Empty's zeros are non-negative, so for an MspDB result it answers TRUE for exactly the candidates that have to be excluded -- inert as a guard here, and pinned as inert by a test, so that simplifying to it later says what breaks. The two branches held separate ms2Result locals, so one is hoisted out. LipidMs2MatchResult derives from Ms2MatchResult and carries the same flag, and its Empty still has an empty Name and IsOtherLipidMatch false, so the lipid branch keeps falling back to the reference's own name. Paired with the evidence ordering in the previous commit. Both halves are needed for the annotation to come out right: the rank says a compared spectrum outranks a bare mass, and this says the bare mass is worth what it measured rather than a third of it. The compared path is unchanged and tested as unchanged -- when all three terms were measured the divisor is still three and the value is identical. Only candidates that had terms fabricated for them move. Reverting the change fails three of the six new tests. DI-MS is the mode the author has said has few users and least development; this is a score change and it is deliberate. 1,756 passed, 0 failed, 17 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two in-silico members tied with PrecursorOnly because the question had
not been put. The author answered it on 2026-09-15, and the reasoning is
worth more than the placement.
An in-silico tool takes the product-ion spectrum into account. MS-FINDER
and SIRIUS read the measured peaks to get where they get; a spectrum
predictor is scored against them. So the precursor mass is the one term an
in-silico candidate and a precursor-only candidate have in common, and
everything else the calculation used is extra.
And the precursor-only candidate is making the larger claim. It arrives as
a STRUCTURE, a named compound, on evidence that justifies at most a
formula -- "それは基本行き過ぎ". Retention time plus m/z would be another
matter; m/z alone is not. What ranks below here is therefore not an
absence of evidence but a claim that overreaches its evidence, which is a
different and worse thing.
compared, carried the match 6 ReferenceSpectrum, RuleBased
compared, fell short 5 WeakSpectrumMatch
not recorded / asserted by a person 4 Unspecified, Manual
calculated 3 the two in-silico members
mass alone 2 PrecursorOnly
compared, explained nothing 1 UnmatchedSpectrum
Unspecified stays between the compared tiers and the calculated one, which
is what "neither credited nor penalised" now means, and a project in which
nothing is recorded still has every candidate tied.
In-silico remains below WeakSpectrumMatch, and that is not in tension with
the above: a real spectrum that partly agreed is an observation of the
compound, a calculation is a hypothesis about it. Both bounds are tested.
No interaction with MS-FINDER acceptance in the GUI, which sets
SourceType.Manual and therefore wins at the first key of both orderings
before this one is read.
The numbers are ordinals and nothing serializes them, so the renumbering
costs nothing. Tying the two back together fails three tests.
1,758 passed, 0 failed, 17 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"An experimentally acquired reference spectrum was compared" was asserted unconditionally, in two places, about every spectral library MS-DIAL has ever searched. Nothing checked it, and for a growing share of libraries it is false: NEIMS generates EI spectra for GC-MS, CFM-ID and its kin generate MS/MS, and MS-DIAL parses, searches and scores those exactly as it does Wiley, NIST or MassBank. The author asked on 2026-09-15 for the two to be tagged apart. MS-DIAL cannot tell by looking -- an MSP holds no field that says so -- so the analyst says which, by choosing DataBaseSource.PredictedMsp instead of Msp when adding the library, and every annotator reads it off the database it is already holding. The two evidence members this needs have existed since the in-silico vocabulary commit; what was missing was anything able to choose between them. ForDatabaseMatch now takes the database, and requires it. That is what found the sites: the compiler listed all six, including the GC-MS funnel, which had been asserting ReferenceSpectrum in its own separate line of code. It now goes through the same function, so the claim is made in one place for every mode. A future annotator has to answer the question rather than inherit "acquired" by saying nothing. The order of the three questions is deliberate. What was compared comes first, because a predicted library says nothing about a feature that had no spectrum. The lipid rule set comes second, because there the diagnostic-fragment rules are the evidence whatever the spectra were. The library kind is last. DataBaseSource.Lbm is deliberately NOT counted as predicted, even though .lbm2 spectra are in silico, and the reason is a defect rather than a principle: MassAnnotationSettingModel.LoadDataBase switches on DBSource and then passes the Lbm constant regardless, so a plain MSP library can arrive wearing that label. Understating is better than mislabelling those. The EAD lipid databases are counted, because MS-DIAL builds them itself. DataBaseSettingModel.LoadMspDataBase had the same shape of bug in miniature -- it passed the DataBaseSource.Msp constant rather than the DBSource it had just switched on, discarding the analyst's answer at the one point it was known. Fixed here because the feature does not work otherwise. PredictedMsp is appended, not inserted: the value is serialized in MoleculeDataBase key 2 and in the restoration keys, so the existing members keep the numbers projects on disk already hold. It is read exactly like Msp everywhere -- same parser, same annotator, same default cut-offs -- and the GC-MS test asserts the score and the verdicts are identical between the two. The only difference is the record, and through the evidence rank, that an acquired match wins a tie. Not done here, and not a side effect worth taking on quietly: MethodSettingModelFactory's four "the analyst already added an MSP" guards still test Msp alone. They govern whether a lipidomics run auto-attaches a bundled LBM, which is a different question from this one. Reverting the tag fails five tests across the two suites. 1,771 passed, 0 failed, 17 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The library-kind tag shipped reachable only from the GUI dropdown. Every run the reanalysis pipeline makes is a Console run, and the Console built DataBaseSource.Msp unconditionally -- so an in-silico library given to the pipeline would have been published as a reference-spectrum match, which is the assertion this whole line of work exists to stop making. The MSP annotator settings TSV already carries per-annotator columns for priority, target omics and the search tolerances, so this is one more: library_kind (or library_type, spectra_source, msp_kind), reading acquired / experimental / measured / msp against predicted / in silico / computed / generated. Silence means acquired, so an existing settings file runs exactly as it did. An unrecognised value is reported and treated as silence. Failing the run would cost a whole reanalysis for a misspelt word; guessing "predicted" would publish an in-silico claim nobody made. The run log states the kind each annotator actually used, next to the tolerances it already states. A file is one library, so the kind belongs to the file group rather than to each annotator over it. Two annotators on the same file that disagree is a mistake in the settings file, and it is named in the log rather than silently resolved. Scope, stated rather than left to be discovered: only LcmsProcess takes the multi-annotator overload. DIMS, GC-MS, IMMS and LC-IM-MS still go through the single-library ParseLibraries, which has no settings table to read and so still builds DataBaseSource.Msp. LC-MS is the whole of the public repository campaign, so the pipeline is covered; GC-MS -- where NEIMS raised the question -- is not, and giving it one needs either a method-file key backed by a new ParameterBase field or the same settings table, which is a decision rather than a detail. Always returning Msp fails two of the three new tests. 1,774 passed, 0 failed, 17 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…the spectrum Peak table, alignment table, mzTab-M and exported spectra are joined by peak ID, and that contract already held. What did not is the field a person looks at first. mzTab-M alone canonicalised the name; the two tables and every spectral export published spot.Name raw, so the same peak read "low score: Quercetin" in the CSV and "Quercetin" in the mzTab -- and a reader has no reason to suspect that one exporter strips a prefix and another does not. AnnotationName.Canonical is now the single rule, applied at every exit: strip the processing prefix, reduce a dual lipid name to the level the evidence supports. What the prefix used to say travels as its own field, where it says more than the prefix could -- "Evidence source" and "Measured terms" in the tables, opt_global_evidence_source in mzTab-M, EVIDENCE= in a spectrum's COMMENT. The prefix collapsed "compared and fell short" and "compared and explained nothing" into one word and could not name the library at all. The structural level still comes from the match record, never from the text. "PC 34:1|PC 16:0_18:1" means chains resolved when MsScanMatching wrote it and chains unsupported when MsReferenceScorer did; reading the string cannot tell them apart. So the feature is passed rather than the flag -- AnnotatedObjectExtensions.CanonicalName -- and a caller cannot pair one feature's name with another's verdict, or forget the verdict and publish an sn-chain composition that nothing measured. The exported spectra now carry the evidence record in COMMENT, beside PEAKID: EVIDENCE= and TERMS=. That is what the join was for. A spectrum leaves MS-DIAL for MS-FINDER, ICEBERG or SIRIUS and comes back as a new annotation, and until now what MS-DIAL had already established about it stayed behind in a table -- the downstream tool could not see whether the spectrum arrived already identified by a reference match or entirely unexplained. A peak with nothing recorded says "null" rather than dropping the fields, so the field set is stable enough to parse. THE TERM LIST USES A COMMA THERE, NOT A PIPE. COMMENT is a '|'-delimited list of key=value fields, and the tables and mzTab-M join measured terms with '|'. Reusing it here would turn one value into several fields and a reader splitting on '|' would lose every term after the first, with no error -- the exact silent loss this record exists to end. The tab-delimited formats have no collision and keep the '|' form. The change is visible in the CSV: a suggestion that used to read "w/o MS2: PC 34:1" now reads "PC 34:1", with "Evidence source" two columns along. That is the same decision already taken for mzTab-M, extended to the other three because the author asked for the four to agree. Publishing the raw name again fails two of the five new tests. 1,779 passed, 0 failed, 17 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…y it used Two defects an audit of Peak ID traceability turned up, both of which end with artifacts describing a run that did not happen. THE WORKER POOL. The Console sized it as NumThreads / 2, integer division, no floor, in all five modes -- while every GUI path uses Math.Max(1, UsableNumThreads / 2). So "number of threads: 1", a value the method-file parser accepts, gave zero workers. ProcessRunner then built an empty Task array and RunAllAsync returned an already-completed task having peak-picked, deconvoluted and annotated nothing. The empty run was not the damage. MS-DIAL writes its per-file intermediates beside the raw data it reads, so a re-analysis of data processed before found the earlier run's .pai files, loaded them, and exported THE PREVIOUS RUN'S PEAKS AND PEAK IDS -- under this run's parameter file, library list and version string. Peak ID is the join key across .mdpeak, .mdmsp, .mdalign, .mzTab and .mdpeakid.tsv, so the whole artifact set would agree with itself about a run nobody performed. With no pre-existing .pai it crashes instead, which is the good case. Fixed at the five call sites to match the GUI, and ProcessRunner now REFUSES a non-positive worker count rather than clamping it. Clamping would hide the caller's arithmetic error; every caller already floors its own value, so this is the backstop that speaks up when one stops. A silent no-op that reports success is the failure mode worth making impossible. The hazard was newly introduced: the commit that made "number of threads" readable from a method file is the commit that created the zero case. THE UNDECLARED LIBRARY. DataBaseSource.PredictedMsp, added two commits ago, has no case in MztabFormatExport.SetDatabaseList -- so an in-silico library vanished from the mzTab-M metadata section while its annotation rows kept citing its database[n] prefix. The file referred to a database it never defined. My omission, and the shape of it is the one this branch keeps finding: a switch that was complete when it was written and silently is not after a member is added. It is declared separately from Msp rather than folded in, because telling a generated library from an acquired one is the entire purpose of the kind. The test enumerates the library kinds the Console can build and fails by name when one produces no database block, so the next addition is caught at the point it is made. Both mutations checked: removing the case fails two tests, accepting zero workers fails one. 1,784 passed, 0 failed, 17 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A public library is citable: the Zenodo DOI beside it in a run's provenance
identifies it exactly and anyone can fetch it. A laboratory's own MSP has no
such handle, and what a run records about one is its absolute path -- exact
for the person holding that disk, useless to every reader.
The author's framing on 2026-09-15: what one would want is for the record
count, the compound count and similar metadata to be publishable on their
own, so a private library could be registered by its statistics rather than
by its contents. No registry exists for that. MS-DIAL can at least emit the
statistics, which is the precondition for anyone ever registering them, and
which is already enough for a reader to check that two runs searched the
same library and for a laboratory to recognise its own.
MTD custom[1] [,, MS-DIAL library statistics database[1],
182345 records; 31204 compounds; sha256:91144a524bc07672]
COMPUTED FROM THE STORED RECORDS, NOT FROM THE FILE. The project keeps its
own copy of every reference it loaded, so the digest describes the library
the run ACTUALLY SEARCHED and stays answerable when the original file has
moved, been renamed, or been edited since. A checksum of whatever is at that
path today answers a weaker and different question.
Derived rather than stored, for the same reason the evidence rank is a
switch and not a cast: MoleculeDataBase has some 220 construction sites, so
a stored field is one any of them could forget, get wrong, or let fall out
of step with the records beside it. It cannot drift if it is computed from
them. This also means no new MessagePack key and no project-format change.
Record count and compound count answer different questions and the gap
between them is the informative part: many adducts and collision energies
per structure is a different kind of library from a flat list, at equal
record count. Compounds collapse on the InChIKey skeleton block, which is
what that block is for; records with no key count one each, because a
library of ten thousand unidentified spectra does not hold one compound.
Intensities are excluded from the digest on purpose -- several load paths
normalise them, so including them would make the identity depend on how the
library was read rather than on what it holds, and two projects that loaded
the same file by different routes would disagree about which library they
used. m/z is rounded to five decimals for the same reason.
database[n]-uri is untouched: the author has said the absolute path there is
not a problem for him.
Every identity-bearing field is tested as changing the digest, and rescaling
every intensity is tested as not.
1,791 passed, 0 failed, 17 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MS-DIAL 5 stated its version from three hand-maintained .resx literals that
had drifted apart and gone stale: the GUI said 5.5.250403-beta, the Console
said 5.5.241113 -- frozen since 2024-11-19 while two hundred commits landed
-- and MsdialCore, from which ParameterBase takes its default, still said
4.24. So every build since November 2024 reported the same version, and
someone who ran a locally built MS-DIAL for a paper could not afterwards say
which one. That is the author's complaint of 2026-09-15.
MainVersion 5.5 one line in Directory.Build.props
BuildDate 2026-09-15 stamped at build time
CommitId d1ba6b5 already compiled in; see below
DisplayVersion 5.5.260915 what a user sees
FullIdentity 5.5.260915+d1ba6b59 what provenance carries
The split is what he asked for: main version plus date for display, commit
kept internal so the user-facing string stays short. A window title is not
where a hex string earns its space; an exported file is exactly where it
does, because two builds made on one day share a display version and nothing
about it identifies the source.
THE COMMIT ID WAS ALREADY THERE. SourceLink is enabled by the .NET SDK for
every SDK-style project in this tree with no configuration at all, and the
generated AssemblyInfo.cs has carried
AssemblyInformationalVersion = "1.0.0+<40 hex>" all along. Nobody read it.
So this needs no git at build time, no network at run time, and no new
build dependency -- only the date, which is two AssemblyMetadata items.
Read from CommonStandard's own assembly, deliberately. Everything references
it and they are built from one working tree, so its commit is theirs. It
also sidesteps MsdialGuiApp setting GenerateAssemblyInfo=false, which leaves
the GUI with no informational version of its own: asking the entry assembly
would return nothing exactly where the answer is most wanted.
IsNewerThan is the question the update notification needs and never asked.
It compares version strings for INEQUALITY, so a locally built MS-DIAL is
told on every start that a new version is available and the newer thing is
the one already running. The release feed already carries published_at and
MS-DIAL already parses it into VersionDescriptionDocument.DatePublished,
where it has never been read once. Comparison is by day, because the
resolution of a build stamp is a day; same-day is not newer, so a release
published the day a build was made still notifies. An unparseable or absent
date returns false, leaving the existing behaviour alone -- returning true
there would silently suppress every notification the moment GitHub changed
its date format, and nobody would notice for a year.
Every value degrades to "unknown" independently rather than throwing. A
version string is not worth failing a run over, and a partial identity is
worth more than none.
This commit adds the identity and proves the stamping works in this build.
Wiring it into the GUI, the Console and the exported provenance -- and
retiring the three literals -- is the next one, because it changes what
users see and what files say.
1,797 passed, 0 failed, 17 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The update check compared version STRINGS for inequality. A locally built
MS-DIAL never equals the published tag, so everyone developing MS-DIAL --
and anyone running the build they made for a paper -- was told on every
start that a new version was available, and the newer thing was the one
already running. The author raised it on 2026-09-15 as a daily annoyance.
The release feed has carried published_at since this class was written, and
MS-DIAL has parsed it into VersionDescriptionDocument.DatePublished and
never read it once. So the fix is a guard, not a feature: if this build is
newer than the release, say nothing.
Same-day still notifies and an unparseable date still notifies. The cost of
a redundant dialog is one click; the cost of the opposite error is a user
sitting on an old version for a year without knowing, which is the failure
nobody would ever report.
ALSO A TRAP DEFUSED. LatestVersion came from
TagName.TrimStart("MSDIAL-v".ToCharArray()), which trims ANY of
{M,S,D,I,A,L,-,v} repeatedly rather than the prefix as a unit. It gives the
right answer today only because the release selection twenty lines above
requires the tag to start "MSDIAL-v5", so the trim stops on a '5' that is
not in the set -- accidental safety resting on a guard in another method.
A tag whose version part began with any of those characters would have had
them eaten off the front, and since a mangled version never equals the local
one, the dialog would then have appeared on every start forever. Now a
prefix strip, and tested with the versions that would have been eaten.
MSDIAL4 has the same TrimStart idiom with the same accidental guard. Left
alone: MS-DIAL 4 is maintenance-only and its tag scheme is not changing.
Not in this commit, because it changes what users see and needs the author's
call: retiring the three stale .resx literals in favour of
MsdialBuildIdentity.DisplayVersion. GlobalResources.IsLabPrivate derives a
BEHAVIOURAL flag from the version string's suffix -- it tests for "-dev" and
"-tada", and it gates both this update check and several GUI features -- so
changing the version's shape silently changes which features a build has.
How a laboratory build should be marked once the version is generated is a
decision, not a detail.
1,801 passed, 0 failed, 17 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
before MTD software[1] [MS, MS:1003082, MS-DIAL, Msdial console 5.5.241113]
after MTD software[1] [MS, MS:1003082, MS-DIAL, 5.5.260915+cd5ff71c]
Verified on the fast-LC demo data, not just in tests: the Console banner now
reads "MSDIAL Console Application 5.5.260915" and the exported mzTab names
the commit that produced it.
The three literals are gone, resource entry and generated accessor both, so
nothing can reach them again:
MsdialCore 4.24 ParameterBase's default -- see below
MsdialGuiApp 5.5.250403-beta the GUI's display and update comparison
Console 5.5.241113 the Console banner and its provenance
MsdialCore's "4.24" was the one that mattered. It was
ParameterBase.MsdialVersionNumber's default, so any path that created a
parameter and never assigned it published "4.24" as the MS-DIAL version --
into the project file and into mzTab-M's software entry. The GUI and the
Console both assign it, so it was not reaching a normal run, but a default
that is quietly wrong is a trap waiting for the next caller. It is now the
running build, which is the true answer for a parameter this build just
made. A parameter DESERIALIZED from a project still keeps its stored value,
as it must: that says which build made the project, not which is reading it.
The Console no longer prefixes "Msdial console". The field is the VERSION,
and mzTab-M already wraps it as "MS-DIAL, <this>" -- the old line said the
application twice and the version once, staleness included.
Display gets DisplayVersion and provenance gets FullIdentity, which is the
split the author asked for. --version prints the full identity deliberately:
it is what someone runs to record which build they used, and the commit is
the half that answers that exactly.
THE LABORATORY SUFFIX IS KEPT, not removed. GlobalResources.IsLabPrivate
tests the displayed version for "-dev" and "-tada" and gates several GUI
behaviours on it, including whether the update check runs. The author said
not to worry about it and that IsLabPrivate=false is right for a
repository-reanalysis run -- which it now is, by default. But deleting the
literal would have deleted the mechanism, not just its current value, and
taken away the main developer's way of marking a build. So the suffix moved
to MsdialVersionSuffix in Directory.Build.props: empty by default, and a
laboratory build sets it there instead of editing a checked-in resource and
remembering to change it back.
MSDIAL4 and MSFINDER keep their own literals. MS-DIAL 4 is maintenance-only.
1,801 passed, 0 failed, 17 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every dispatcher read a line, handed it to the parameter readers, and threw
away the boolean saying whether anything had claimed it -- under a comment
reading "// write something if needed". GC-MS, IMMS and LC-IM-MS threw it
away twice, once for the common reader and once for their own.
So a misspelt key, a key from a newer MS-DIAL, or a key copied from another
mode's template was read, matched nothing, and vanished. The run used the
built-in default and said nothing, and the analyst had every reason to
believe their value had been applied -- with a method file that appears to
document the run correctly. The author called this critical on 2026-09-16.
MEASURED BEFORE DECIDING WHAT TO DO ABOUT IT. Run against MS-DIAL's own
shipped lipidomics template, this reports TWENTY-SEVEN keys with no effect:
Only report top hit for LBM-based annotation
Sigma window value
Process option
Replace true zero values with 1/2 of minimum peak height over all samples
Considering Br and Cl for isotopes
Max isotopes detected in ms1 spectrum
Ionization, adduct list
... and the eighteen export switches
Several of those change results, not just output shape. So reporting is what
this commit does and failing is what it must not do: rejecting a method file
with an unread key would reject every method file in existence, MS-DIAL's own
templates first. Making those keys work is a separate decision, because
settings ignored for years would start taking effect and change results.
A BLANK VALUE IS NOT AN ERROR and is reported apart from the rest.
"Msp file path:" with nothing after it is how a method file says there is no
MSP library, and how the templates are written. It is still named, because
from the analyst's side it looks identical to a value that failed to apply.
The molecular-networking reader is deliberately exempt: it is handed the same
method file as the mode reader and claims only its own subset, so reporting
there would list every other key in the file. The mode reader is where a key
gets its verdict.
A file whose keys are all understood prints nothing, and that is tested too:
a report that fires on every run is one nobody reads, which is how
twenty-seven of them stayed invisible.
Discarding the verdict again fails two of the three new tests.
1,804 passed, 0 failed, 17 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
$ MSDIALCUI.exe --version
5.5.260916
commit 4fb297b
MS-DIAL Interactive learns which build it ran by executing this and matching
/(?:^|\s)(\d+\.\d+(?:\.\d+)+)(?:\s|$)/ against the output. That is how
"5.5.241113" reached its publication report -- and why that report said
5.5.241113 for two years. It has no knowledge of its own; it prints back what
MS-DIAL tells it.
So making --version print "5.5.260916+4fb297ba" would have broken its first
read: the '+' is neither whitespace nor end of input, so the pattern misses
and the code falls through to launching the executable a second time and
matching the help banner. Still correct, but at the cost of a process and of
the commit, which would never reach the report at all.
Version on line one, commit on line two. The newline is the whitespace the
existing pattern wants, so nothing downstream changes and what it captures
becomes true instead of stale. Verified by running the built Console and
applying that regex to its real output.
The shape is now pinned by a test, because the constraint is invisible from
inside this repository: the thing that would break lives in another one.
1,805 passed, 0 failed, 17 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An annotation evidence record for MS-DIAL 5: what was compared, what the
comparison was worth, which library it was against, and which build produced it.
26 commits, each with its reasoning in the message and a mutation check.
1,805 tests pass, 0 fail, 17 skip.
The record
Two members on
MsScanMatchResult, written where the facts are known:MeasuredTerms— which similarity terms were actually computed, recorded atthe measurement site rather than inferred later from a value.
IsSpectrumMatchand its siblings say a term passed a threshold; a term that was never computed
also reports false there, and conflating the two is what this ends.
AnnotationEvidenceSource— nine members covering what the evidence was, fromReferenceSpectrumdown toUnmatchedSpectrum.Plus the candidate population: how many were found, passed threshold, and were
reference-matched, so a reader can tell a discarded fourth candidate from there
having been only three references in the library.
Defects this found and fixed
GetGaussianSimilaritydoes not guard, so a library record with no RT scoredagainst −1. Fixed, with the match caps the author set (2 min; 150 Kovats /
50,000 Fiehn).
DimsMspAnnotatoraveraged a fixed three terms, and a feature with no product-ion spectrum got two
fabricated zeros from
Ms2MatchResult.Empty— so a precursor mass worth 1.0 waspublished as 0.33, below any cutoff above a third.
precursor mass alone could beat a name a spectrum decided, on the strength of a
larger number. An evidence rank now sits above the score in all three ordering
keys — an order, not a weight, so it cannot be bought at any score.
NumThreads / 2withno floor gave zero workers at
number of threads: 1, and the export stage theneither crashed or — when the raw data already carried
.paifiles from anearlier run — silently exported the previous run's peaks and Peak IDs under
this run's parameters, library list and version string.
Only report top hit for LBM-based annotation,Sigma window valueandReplace true zero values with 1/2 of minimum peak height. Reported now, notfatal — failing would reject every method file in existence, MS-DIAL's own
templates first.
One name in all four files
The peak table, the alignment table, mzTab-M and the exported spectra are joined
by peak ID, and that held. What did not is the field a person reads first: mzTab-M
alone stripped the
low score:/no MS2:prefixes.AnnotationName.Canonicalis now the single rule at every exit, and what the prefix used to say travels as
its own field — where it says more, since the prefix collapsed "compared and fell
short" with "compared and explained nothing".
Exported spectra now carry the evidence beside
PEAKIDinCOMMENT, so aspectrum handed to MS-FINDER, ICEBERG or SIRIUS no longer leaves behind what
MS-DIAL had already established about it.
Which build, and which library
MsdialBuildIdentityreplaces three hand-maintained.resxliterals that haddrifted apart and gone stale — the GUI
5.5.250403-beta, the Console5.5.241113frozen since November 2024 while two hundred commits landed, and MsdialCore still
4.24, which wasParameterBase's default. Main version plus build date fordisplay, commit for provenance; the commit id was already compiled in by
SourceLink and nobody had read it. The update dialog no longer tells a developer
their newer build is out of date.
A library that no DOI names can now say what it is without saying where it lives:
record count, compound count and a digest of the records the run actually
searched, published in mzTab-M
custom[n].