Making Freesurfer FAIR

Script to Export Freesurfer-based Parcellation/Segmentation Stats and Provenance as JSON-LD and NIDM

Project Description

This project ultimately aims to facilitate both query and analysis of parcellation/segmentation based regional statistics across popular softwares such as Freesurfer, FSL, and ANTS. Currently each software produces its own output format and brain region labels are specific to the atlas used in generating the regional statistics. This makes life difficult when trying to search for "nucleus accumbens" volume, for example, across the different software products. Further, knowing which version of the software tool used and what atlas and version of the atlas in a structured representation facilitating query is lacking. To this end we propose augmenting the various segmentation tools with scripts that will: (1) map atlas-specific anatomical nomeclature to anatomical concepts hosted in terminology resources (e.g. InterLex); (2) capture better structured provenance about the input image(s) and the atlases used for the segmentation; (3) export the segmentation results and the provenance as either JSON-LD, NIDM which can then link the derived data to broader records of the original project metadata, or as an additional component of a BIDS derivative.

We aim to tackle this problem in steps. For this hackathon project we'll be focusing on conversion from Freesurfer's mri_segstats program output along with some additional parsing/conversion of Freesurfer log files. The conversion is driven by a function which queries InterLex and develops a JSON structure which defines the atlas terminology and the measures being output.

Skills required to participate

Python and structural neuroimaging experience. If one has experience with rdflib or PROV that would also be helpful.

Participants

• David Keator
• Adina Wagner
• Jeffrey Grethe
• Satra Ghosh
• David Kennedy
• JB Poline

Integration

This project will need expertise in programming, structural neuroimaging, and anatomy. To make this project sucessful we need individuals who have skills in any of these domains to help with: (1) understand Freesurfer's segmentation results format and log files; (2) programming up a script in Python; (3) understand anatomy well enough to select the proper anatomical concept that maps to a specific atlas designation of a region and can define new anatomy terms where needed, linking them to broader concepts to facilitate segmentation results queries across softwares.

Preparation material

• Freesurfer
• mri_segstats
• rdflib
• PyNIDM
• InterLex Term Search
• Anatomical Term Mappings via InterLex
• Examples done pseudo-manually of Freesurfer, FSL, and ANTS segmentation data added to NIDM documents for ABIDE and ADHD200 datasets

Installation

$ conda create -n segstats_jsonld python=3
$ source activate segstats_jsonld
$ git clone https://github.com/ReproNim/segstats_jsonld.git
$ cd segstats_jsonld
$ pip install -e .

Usage

The program is installed as segstats2nidm; python -m segstats_jsonld.fs_to_nidm runs the same program. Give exactly one of -o (write new NIDM files) and -n (add the stats to an existing NIDM file). With -s, -o is a directory that receives one file per stats file; with -f or -csv it is the output file. The text below is the program's --help.

-csv requires PyNIDM 5. Under PyNIDM 4.5 every -csv run fails inside PyNIDM: its DD_to_nidm adds an rdflib.Namespace as a triple's object, which the rdflib 6 that PyNIDM 4.5 requires refuses ("must be an rdflib term"). -s and -f work with PyNIDM 4.5 and 5.

usage: fs_to_nidm.py [-h] (-s SUBJECT_DIR | -f SEGFILE | -csv CSVFILE) [-subjid SUBJID]
                     (-o OUTPUT | -n NIDM_FILE) [-j] [-add_de] [-forcenidm] [-json_map JSON_MAP]
                     [-acq_filename ACQ_FILENAME [ACQ_FILENAME ...]]

This program reads FreeSurfer stats files (e.g. aseg.stats), maps the FreeSurfer regions and
measures to common data elements, and writes the statistics as NIDM (Turtle or JSON-LD), either as
new files (-o) or added to an existing NIDM file (-n).

options:
  -h, --help            show this help message and exit
  -s SUBJECT_DIR, --subject_dir SUBJECT_DIR
                        Path to Freesurfer subject directory
  -f SEGFILE, --seg_file SEGFILE
                        Path or URL to a specific Freesurfer stats file. Check supported_files for
                        list of supported stats files.
  -csv CSVFILE, --csv_file CSVFILE
                        Path to CSV file which includes a header row with 1 column containing
                        subject IDs and the other columns are variables indicating the Freesurfer-
                        derived region measure (e.g. volume, surface area, etc. If you use this
                        mode of running this software you will be asked to match your region
                        variables to standard Freesurfer region / measure labels presented to you
                        (so this program will end up being interactive. In future WIP we will add
                        the ability to export a json sidecar file with those mappings for
                        automated runs of future CSV files with the same set of variables.
  -subjid SUBJID, --subjid SUBJID
                        Subject identifier to be used in the NIDM files. Required with
                        -f/--seg_file. With -s/--subject_dir it defaults to the name of the
                        subject directory, and when given it overrides the subject directory name
                        (set it when that directory is not named after the subject, e.g.
                        sub-01/freesurfer). Earlier releases ignored -subjid with -s and always
                        used the directory name. Ignored with -csv, where the ID column names each
                        subject
  -o OUTPUT, --output OUTPUT
                        Where to write a new NIDM file. With -s, the directory that gets one
                        <stats>_nidm.ttl (or .jsonld) per stats file; with -f or -csv, the output
                        file. Exactly one of -o/--output and -n/--nidm is required
  -n NIDM_FILE, --nidm NIDM_FILE
                        Existing NIDM file to add the stats to; it is rewritten in place (with -j
                        the result goes to <NIDM_FILE>.json instead). Exactly one of -o/--output
                        and -n/--nidm is required
  -j, --jsonld          If flag set then NIDM file will be written as JSONLD instead of TURTLE.
                        With -n the JSON-LD goes to <NIDM_FILE>.json. Not allowed with -s and -n
                        together
  -add_de, --add_de     If flag set then the data element dictionary is added to the NIDM file,
                        else it is written to a separate file, fs_cde.ttl, in the output directory
                        (or the directory of the -n NIDM file)
  -forcenidm, --forcenidm
                        With -n: if the NIDM file has no subjects at all, add a new subject agent
                        with the subject ID and attach the stats to it. If the file has subjects
                        and none matches the subject ID (compared without a leading "sub-" and
                        leading zeros), the run fails with or without this option
  -json_map JSON_MAP, --json_map JSON_MAP
                        If storing freesurfer segmentation data stored in a CSV file and you've
                        previously stored the variable to Freesurfer CDE mappings, this can be
                        reused. Otherwise this argument is ignored
  -acq_filename ACQ_FILENAME [ACQ_FILENAME ...], --acq_filename ACQ_FILENAME [ACQ_FILENAME ...]
                        With -s or -f and -n: file names (or paths) of the anatomical images the
                        stats were computed from (several for repeated runs). The stats are linked
                        to the input NIDM acquisitions with those file names if each name matches
                        exactly one of the subject's anatomical acquisitions, and left unlinked
                        otherwise. Without it the stats are linked only when the subject has a
                        single anatomical acquisition

Changes in behaviour from earlier releases

• -subjid with -s now sets the subject ID, overriding the subject directory name. Earlier releases ignored -subjid with -s and always used the directory name, so a caller that passes both now gets the -subjid value as ndar:src_subject_id (for example ds000001/sub-01 instead of sub-01). Drop -subjid to keep the directory name.
• With -n, the subject is found by comparing IDs without a leading sub- and leading zeros on both sides. If the NIDM file has subjects and none matches, the run fails (exit 1) and writes nothing, with or without --forcenidm; earlier releases attached the stats to a new subject agent under --forcenidm. If several match and all store the same ID, the one that owns an anatomical MRI acquisition is used (the run prints which); if none or more than one of them does, the run fails and writes nothing. If several match but store different IDs (sub-01 and 001), the run fails and writes nothing. --forcenidm now adds a subject only to a NIDM file that has none, and without it such a file is an error. A subject is any node with ndar:src_subject_id (under the https:// or the http:// NDAR IRI), whatever its rdf:type. The lookup is segstats_jsonld.fs_to_nidm.find_subject(graph, subject_id, forceagent=False, created_agents=()), which returns the subject node (or None for a file without subjects under forceagent), raises SubjectLookupError otherwise, and never changes the graph; ants_seg_to_nidm has the same function with the same rules.
• -s with no supported stats file in its stats/ directory fails (exit 1) with a message naming the directory and the files looked for; earlier releases exited 0 without writing anything.
• -j with both -s and -n is refused with a message naming the alternatives (earlier releases failed on it with an rdflib plugin error).

Updating the FreeSurfer Common Data Elements

This section describes how one can update the common data elements from FreeSurfer as they update the names they are using for their elements. Run it from the repository root with the package installed in editable mode (pip install -e .), so the files under segstats_jsonld/mapping_data/ are the ones written.

from segstats_jsonld.fsutils import collate_fs_items, create_fs_mapper, create_cde_graph
collate_fs_items('/path/to/statsdir', error_on_new_key=False)
create_fs_mapper()

This adds new structures to freesurfer-cdes.json and freesurfermap.json. You can edit freesurfermap.json to improve connections to ontologies (e.g., by editing the isAbout) and run create_fs_mapper() again to carry the edits into freesurfer-cdes.json.

Then regenerate fs_cde.ttl, the file PyNIDM reads, from freesurfer-cdes.json:

create_cde_graph().serialize(destination="segstats_jsonld/mapping_data/fs_cde.ttl", format="turtle")

tests/test_fs_cde_ttl.py fails while fs_cde.ttl and freesurfer-cdes.json disagree.

PyNIDM (4.x and 5.x) caches the data elements it loads in <temp dir>/cde_graph.<hash>.pickle (<temp dir> is Python's tempfile.gettempdir(), usually $TMPDIR or /tmp) and reuses that file without checking whether the .ttl files changed. After fs_cde.ttl changes, delete cde_graph.*.pickle there, or PyNIDM keeps resolving against the old data elements.