Skip to content

Add BIDS sanitize_label(); use it to match file-name labels to metadata identifiers - #447

Draft
yarikoptic-gitmate wants to merge 1 commit into
masterfrom
bids-label-sanitization
Draft

yarikoptic-gitmate wants to merge 1 commit into
masterfrom
bids-label-sanitization

Conversation

@yarikoptic-gitmate

@yarikoptic-gitmate yarikoptic-gitmate commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • BIDS 1.11.0 allows + in labels (bids-standard/bids-specification#1926). This adds dandischema.utils.sanitize_label() and BIDS_LABEL_REGEX. The function replaces every character other than an ASCII letter, an ASCII digit or + with +, one character at a time, so fly_226 becomes fly+226. It is meant to be the single definition shared with dandi organize (dandi/dandi-cli#1981).
  • sanitize_value() replaces only characters that sanitize_label() replaces too, so sanitize_label(sanitize_value(v)) == sanitize_label(v) for any v. A label produced by any dandi-cli version and the identifier it came from therefore map to the same value.
  • aggregate_assets_summary() uses that property. A sub- or sample- label from a file name now adds a subject or sample only if its sanitize_label() form differs from that of every identifier in the metadata. Before, the raw label was compared with the sanitize_value()-ed identifier. Names written by older dandi-cli versions, which kept , and spaces, therefore counted each subject twice, and with the new numberOfSubjects <= numberOfFiles assertion that breaks aggregation.
  • sanitize_value() itself is unchanged.

Effect on DANDI

I recomputed aggregate_assets_summary() with master and with this branch for all 606,651 assets of the 1,001 public draft Dandisets, using their dandisets/<id>/draft/assets.jsonld manifests (2026-11-05). Exactly 6 Dandisets change, and all other counts, including numberOfSessions and numberOfSamples, are identical:

Dandiset Files Subjects on master This PR File-name label vs metadata identifier
000053 359 35 34 Ford Prefect vs Ford Prefect (sanitize_value() → Ford-Prefect)
000217 1,121 50 25 Mouse 2054 vs Mouse 2054 (sanitize_value() → Mouse-2054)
000246 2,812 195 186 mouse1-DC1-1L-C2-1R-1 Left vs mouse1_DC1-1L_C2-1R-1 Left
000550 17 AssertionError 17 Fig6-5xBi-0,83MHz-08 vs Fig6_5xBi_0,83MHz_08
000631 15 AssertionError 15 600ns-4kV-0,8MHz-BP-8-24-21-BPAE-20 vs 600ns_4kV_0,8MHz_BP_8-24-21_BPAE_20
000703 22 AssertionError 22 …-nac (2) vs …_nac (2)

The assertion still fails for 001248, 001677 and 001869. Their file names and NWB subject_id genuinely disagree: a placeholder subject_id, R2302 vs R23-02, and fly-226 vs fly_0.

Design notes

  • One character at a time, nothing collapsed or stripped. Collapsing runs and stripping a trailing + would merge distinct subjects in public data: …2000um2_-0.4V vs Eref… and …2000um2_0.4V vs Eref… (000983), and 4484 vs 4484? (001195).
  • Metadata identifiers are not merged by sanitize_label(). 001702 has distinct subjects 5XFAD+_… and 5XFAD-_… (transgene ±). Sanitizing both sides symmetrically would count 67 subjects instead of its 82. Only file-name labels are matched against metadata in sanitize_label() form; everything else is compared as before.
  • Session counting from file names is unchanged.
  • Not changed: the numberOfSubjects <= numberOfFiles assertion. dandi-archive would still fail to publish the 3 Dandisets above, so whether that check should become a warning is a separate decision.

Testing

  • pytest dandischema: 333 passed, 17 skipped. On master: 320 passed. The new tests cover sanitize_label(), the sanitize_value() round trip over all printable ASCII characters, and aggregation for the 000550, 001702 and 001869 cases.
  • codespell, flake8, mypy dandischema and pre-commit (black, isort) are clean.
  • The archive-wide comparison is described above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PvaGRZLHuwDgtWrKMFiMZ8

…adata

BIDS 1.11.0 allows "+" in labels (bids-standard/bids-specification#1926),
so a label can be made BIDS compliant by replacing every character other
than an ASCII letter, digit or "+" with "+".  Add `sanitize_label()` (and
`BIDS_LABEL_REGEX`) doing exactly that, one character at a time, so that
e.g. "-0.4V" and "0.4V" stay distinct.  It is meant to be shared with
dandi-cli's `organize`, so the two produce and compare labels the same way.

Because `sanitize_value()` replaces only characters which `sanitize_label()`
replaces too, `sanitize_label(sanitize_value(v)) == sanitize_label(v)`.
`aggregate_assets_summary()` now relies on that to recognize a `sub-` (or
`sample-`) label in a file name as the identifier from the metadata it was
produced from.  Before, the label was compared as is against the
`sanitize_value()`-ed identifier, so names produced by older dandi-cli
versions, which kept "," or spaces (e.g. `sub-Mouse 2054`, or
`sub-Fig6-5xBi-0,83MHz-08` for "Fig6_5xBi_0,83MHz_08"), counted each subject
twice.  On the 1,001 public draft Dandisets this changes numberOfSubjects
for 6 of them, and the `numberOfSubjects <= numberOfFiles` assertion no
longer fails for 000550, 000631 and 000703.  It still fails for 001248,
001677 and 001869, whose file names and NWB subject_id genuinely disagree.

Identifiers from the metadata are still compared as before, so distinct
subjects such as "5XFAD+_M1" and "5XFAD-_M1" (as in 001702) remain distinct
even though their `sanitize_label()` forms coincide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvaGRZLHuwDgtWrKMFiMZ8
@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.91667% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.08%. Comparing base (eef36dd) to head (a8c80d1).

Files with missing lines Patch % Lines
dandischema/metadata.py 0.00% 10 Missing ⚠️
dandischema/utils.py 0.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #447      +/-   ##
==========================================
+ Coverage   48.56%   49.08%   +0.52%     
==========================================
  Files          19       19              
  Lines        2477     2522      +45     
==========================================
+ Hits         1203     1238      +35     
- Misses       1274     1284      +10     
Flag Coverage Δ
unittests 49.08% <72.91%> (+0.52%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

minor Increment the minor version when merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants