Repository navigation
Add BIDS sanitize_label(); use it to match file-name labels to metadata identifiers - #447
Draft
yarikoptic-gitmate wants to merge 1 commit into
Draft
yarikoptic-gitmate wants to merge 1 commit into
yarikoptic-gitmate wants to merge 1 commit into
Conversation
…adata BIDS 1.11.0 allows "+" in labels (bids-standard/bids-specification#1926), so a label can be made BIDS compliant by replacing every character other than an ASCII letter, digit or "+" with "+". Add `sanitize_label()` (and `BIDS_LABEL_REGEX`) doing exactly that, one character at a time, so that e.g. "-0.4V" and "0.4V" stay distinct. It is meant to be shared with dandi-cli's `organize`, so the two produce and compare labels the same way. Because `sanitize_value()` replaces only characters which `sanitize_label()` replaces too, `sanitize_label(sanitize_value(v)) == sanitize_label(v)`. `aggregate_assets_summary()` now relies on that to recognize a `sub-` (or `sample-`) label in a file name as the identifier from the metadata it was produced from. Before, the label was compared as is against the `sanitize_value()`-ed identifier, so names produced by older dandi-cli versions, which kept "," or spaces (e.g. `sub-Mouse 2054`, or `sub-Fig6-5xBi-0,83MHz-08` for "Fig6_5xBi_0,83MHz_08"), counted each subject twice. On the 1,001 public draft Dandisets this changes numberOfSubjects for 6 of them, and the `numberOfSubjects <= numberOfFiles` assertion no longer fails for 000550, 000631 and 000703. It still fails for 001248, 001677 and 001869, whose file names and NWB subject_id genuinely disagree. Identifiers from the metadata are still compared as before, so distinct subjects such as "5XFAD+_M1" and "5XFAD-_M1" (as in 001702) remain distinct even though their `sanitize_label()` forms coincide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvaGRZLHuwDgtWrKMFiMZ8
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #447 +/- ##
==========================================
+ Coverage 48.56% 49.08% +0.52%
==========================================
Files 19 19
Lines 2477 2522 +45
==========================================
+ Hits 1203 1238 +35
- Misses 1274 1284 +10
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
+in labels (bids-standard/bids-specification#1926). This addsdandischema.utils.sanitize_label()andBIDS_LABEL_REGEX. The function replaces every character other than an ASCII letter, an ASCII digit or+with+, one character at a time, sofly_226becomesfly+226. It is meant to be the single definition shared withdandi organize(dandi/dandi-cli#1981).sanitize_value()replaces only characters thatsanitize_label()replaces too, sosanitize_label(sanitize_value(v)) == sanitize_label(v)for anyv. A label produced by any dandi-cli version and the identifier it came from therefore map to the same value.aggregate_assets_summary()uses that property. Asub-orsample-label from a file name now adds a subject or sample only if itssanitize_label()form differs from that of every identifier in the metadata. Before, the raw label was compared with thesanitize_value()-ed identifier. Names written by older dandi-cli versions, which kept,and spaces, therefore counted each subject twice, and with the newnumberOfSubjects <= numberOfFilesassertion that breaks aggregation.sanitize_value()itself is unchanged.Effect on DANDI
I recomputed
aggregate_assets_summary()withmasterand with this branch for all 606,651 assets of the 1,001 public draft Dandisets, using theirdandisets/<id>/draft/assets.jsonldmanifests (2026-11-05). Exactly 6 Dandisets change, and all other counts, includingnumberOfSessionsandnumberOfSamples, are identical:masterFord PrefectvsFord Prefect(sanitize_value()→Ford-Prefect)Mouse 2054vsMouse 2054(sanitize_value()→Mouse-2054)mouse1-DC1-1L-C2-1R-1 Leftvsmouse1_DC1-1L_C2-1R-1 LeftAssertionErrorFig6-5xBi-0,83MHz-08vsFig6_5xBi_0,83MHz_08AssertionError600ns-4kV-0,8MHz-BP-8-24-21-BPAE-20vs600ns_4kV_0,8MHz_BP_8-24-21_BPAE_20AssertionError…-nac (2)vs…_nac (2)The assertion still fails for 001248, 001677 and 001869. Their file names and NWB
subject_idgenuinely disagree: a placeholdersubject_id,R2302vsR23-02, andfly-226vsfly_0.Design notes
+would merge distinct subjects in public data:…2000um2_-0.4V vs Eref…and…2000um2_0.4V vs Eref…(000983), and4484vs4484?(001195).sanitize_label(). 001702 has distinct subjects5XFAD+_…and5XFAD-_…(transgene ±). Sanitizing both sides symmetrically would count 67 subjects instead of its 82. Only file-name labels are matched against metadata insanitize_label()form; everything else is compared as before.numberOfSubjects <= numberOfFilesassertion. dandi-archive would still fail to publish the 3 Dandisets above, so whether that check should become a warning is a separate decision.Testing
pytest dandischema: 333 passed, 17 skipped. Onmaster: 320 passed. The new tests coversanitize_label(), thesanitize_value()round trip over all printable ASCII characters, and aggregation for the 000550, 001702 and 001869 cases.codespell,flake8,mypy dandischemaand pre-commit (black, isort) are clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01PvaGRZLHuwDgtWrKMFiMZ8