python3 mcin_bidslayout.py -d <bids_directory>

This is a companion script for use with LORIS-MRI.

LORIS-MRI contains a processing pipeline written in Python to assist with the analysis and import of data from BIDS datasets to the LORIS software. These tools depend on a library, pybids, which is used to model a BIDS dataset in an object-oriented way. This library uses Python's os.walk() function to read a BIDS folder. For most cases this works well; however, problems can occur with very large datasets. os.walk() calls os.stat() on each file which increases runtime dramatically.

In the context of a project at MCIN, we had a use case where we needed to remotely mount a large dataset which would be read into a LORIS database. Due to the size of this dataset, pybids was not able to complete the analysis of the directory and the maintainters reported that there would be no quick or easy fix to ameliorate this. (This was exacerbated on our side as the pipeline was running over files mounted remotely via sshfs which increased processing time.)

Our pipeline failed when calling BIDSLayout from pybids. Rather than read all the folders into memory at run-time, we have written this script to read through a large BIDS directory once and to convert its data into a simple CSV file.

This CSV file will be analyzed by LORIS-MRI's Python scripts and read into a format matching the key-value pairs used in pybids's BIDSLayout.

This should address the issue by:

• Eliminating all the file read operations during the processing pipeline. Instead, it will only read from the CSV file.

• Using a generator pattern to read information about one BIDS file at a time and perform actions on it before reading the complete folder.

The CSV file must contain the following headers:

• participant_id
• visit_label
• modality
• scan_type
• nifti_file_path
• json_file_path

As you can see, JSON sidecar files are stored in the same row as their corresponding (NIFTI) image file, rather than storing one file per row.