Convert your fMRI data into BIDS format¶

TODO: [ANDREA] this page is unnecessary complex. I think we should make it simpler by giving the standard steps from DICOM to BIDS, which is in iny case safer than the nift + json editing.

BIDS standards¶

To organize our fMRI dataset, we follow the BIDS Specification.

If you are not familiar with the BIDS Specification, the BIDS Starter Kit provides all the information needed to get started, along with example BIDS datasets, Talks and Slides, and most importantly Tutorials.

It is crucial that you get familiar with BIDS folders/files naming convention and structure. Most, if not all, the tools we are going to use in the next steps are BIDS Apps, and they rely on data organized following the BIDS Specification. Following this structure will make it easier to use these tools, share your code and data, and communicate with other scientists.

The BIDS Specification provides guidelines on how to organize all your data formats, including (f/d)MRI, EEG, eye-tracking, Task events associated with Neuro-Imaging recordings or not, and Derivatives (e.g., pre-processed files, Regions Of Interest mask files, GLM files, etc.).

At any moment, you can check your dataset for BIDS compliance. To do so, you can use the BIDS dataset validator.

BIDS Conversion Overview¶

Here's a high-level overview of the steps involved in arranging your data in a BIDS-compatible way. While this provides a general understanding, most of these steps should be performed using the code provided in each sub-section to minimize errors. After scanning participants, you'll obtain data from two primary sources:

The scanner: functional and structural outputs (.nii files), alongside potential dicom files
The stimulus presentation computer: behavioural outputs (mainly log files and .mat files) and potentially eye-tracking data

As you turn your raw data into a BIDS-compatible format, your project directory will change considerably. The folder trees below show you how each steps will affect your working directory, with changing folders and file in bold for each step.

Step1Step 2Step 3Step 4Step 5Step 6Step 7Step 8Step 9


myproject
└── sourcedata


myproject
└── sourcedata
    └── sub-01
        ├── bh
        ├── dicom
        └── nifti


myproject
└── sourcedata
    └── sub-01
        ├── bh
        │   ├── yyyy-mm-dd-sub-01_run-01_task-taskname_log.tsv
        │   ├── yyyy-mm-dd-sub-01_run-01_task-taskname.mat
        │   ├── ...
        │   ├── yyyy-mm-dd-sub-01_run-xx_task-taskname_log.tsv
        │   └── yyyy-mm-dd-sub-01_run-xx_task-taskname.mat
        ├── dicom
        │   ├── IM_0001
        │   ├── IM_0005
        │   ├── PS_0002
        │   ├── PS_0006
        │   ├── XX_0003
        │   ├── XX_0004
        │   └── XX_0007
        └── nifti
            ├── run-01.nii
            ├── ...
            ├── run-xx.nii
            └── sub-01_struct.nii


myproject
└── sourcedata
    └── sub-01
        ├── bh
        ├── dicom
        ├── dicom_anon
        │   ├── IM_0001
        │   ├── IM_0005
        │   ├── PS_0002
        │   ├── PS_0006
        │   ├── XX_0003
        │   ├── XX_0004
        │   └── XX_0007
        ├── dicom_converted
        │   ├── dcmHeaders.mat
        │   ├── sub-01_run-01.json
        │   ├── sub-01_run-01.nii.gz
        │   ├── sub-01_struct.json
        │   └── sub-01_struct.nii.gz
        └── nifti


myproject
├── BIDS
│   └── sub-01
│       ├── anat
│       │   └── sub-01_T1w.nii
│       └── func
│           ├── sub-01_task-taskname_run-01_bold.nii
│           ├── ...
│           └── sub-01_task-taskname_run-xx_bold.nii
└── sourcedata
    └── sub-01
        ├── bh
        ├── dicom
        ├── dicom_anon
        ├── dicom_converted
        └── nifti


myproject
├── BIDS
│   └── sub-01
│       ├── anat
│       └── func
│           ├── sub-01_task-taskname_run-01_bold.json
│           ├── sub-01_task-taskname_run-01_bold.nii
│           ├── ...
│           ├── sub-01_task-taskname_run-xx_bold.json
│           └── sub-01_task-taskname_run-xx_bold.nii
└── sourcedata
    └── sub-01
        ├── bh
        ├── dicom
        ├── dicom_anon
        ├── dicom_converted
        └── nifti


myproject
├── BIDS
│   └── sub-01
│       ├── anat
│       └── func
│           ├── sub-01_task-taskname_run-01_bold.json
│           ├── sub-01_task-taskname_run-01_bold.nii
│           ├── sub-01_task-taskname_run-01_events.tsv
│           ├── ...
│           ├── sub-01_task-taskname_run-xx_bold.json
│           ├── sub-01_task-taskname_run-xx_bold.nii
│           └── sub-01_task-taskname_run-xx_events.tsv
├── code
└── sourcedata
    └── sub-01
        ├── bh
        ├── dicom
        ├── dicom_anon
        ├── dicom_converted
        └── nifti


myproject
├── BIDS
│   ├── dataset_description.json
│   ├── events.json
│   ├── participants.json
│   ├── participants.tsv
│   ├── sub-01
│   │   ├── anat
│   │   └── func
│   └── task-taskname_bold.json
└── sourcedata
    └── sub-01
        ├── bh
        ├── dicom
        ├── dicom_anon
        ├── dicom_converted
        └── nifti


myproject
├── BIDS
│   ├── .bidsignore
│   ├── dataset_description.json
│   ├── derivatives
│   ├── events.json
│   ├── participants.json
│   ├── participants.tsv
│   ├── sub-01
│   │   ├── anat
│   │   └── func
│   └── task-taskname_bold.json
└── sourcedata
    └── sub-01
        ├── bh
        ├── dicom
        ├── dicom_anon
        ├── dicom_converted
        └── nifti

Your first step is to organize your files in a sourcedata folder. Follow the structure outlined in How to store raw data: have one main project folder (e.g. myproject), and a sourcedata folder in it.
Create the relevant sub-folders within the sourcedata folder: for each participant you collected data from, create a sub-xx folder (e.g. sub-01). Within the folder of each participant, create a bh (behaviour) and nifti (i.e. nifti, the format of the files collected from the scanner) folder. Also create a dicom folder if you collected dicom files for your participant.
Place the files you collected in this sourcedata structure: data collected from your experimental task goes into bh (e.g. .mat files and log files if you used the fMRI task template), data collected from the scanner itself goes in nifti or in dicom based on its format.
If you collected them, proceed to anonymise and convert your DICOM files. Create a dicom_anon and a dicom_converted folder. See below for more details on how to anonimise and convert your dicom files.
Create a BIDS folder in your main project directory, alongside the sourcedata folder. For each participant, create a sub folder (e.g. BIDS/sub-01). In the BIDS folder of each participant, place a func folder for functional files and a anat folder for anatomical files. Copy-paste your functional .nii files from sourcedata to their corresponding func folder, renaming them if necessary to follow BIDS format (e.g. sub-01_task-taskname_run-01_bold.nii), and similarly copy-paste your structural .nii files to the anat folder, renaming them if necessary (e.g. sub-01_T1w.nii). See below for more details on how to rename and move nifti files.
Create .json sidecar files for each functional run .nii file, using the output from the dicom conversion step. If your scanner sequence was the same, your can re-use the same sidecar files across participants (see Creating JSON Sidecar Files below).
Create one events.tsv file for each function run .nii file, using the output from your experimental task. If you used the fMRI task template), output log files can be used to create event files quite easily. More info on events files can be found here:
Create essential modality agnostic BIDS files:
- dataset_description.json
- events.json
- participants.tsv and participants.json
- task-<taskname>_bold.json
Set up additional components:
- Create a derivatives folder for future outputs (e.g. fmriprep output)
- Optional: Include a .bidsignore file if needed

By following these steps systematically, you'll ensure your data is properly organized in BIDS format, facilitating easier analysis and collaboration. Make sure all the steps have been followed successfully by validating your BIDS folder. To do so, use the BIDS validator.

Step-by-step instructions¶

Here we provide more detailed instructions to perform each of the steps mentioned above.

Folder Structure

All the steps and scripts below assume a specific folder structure and file naming convention. They will not work otherwise. Ensure you strictly follow the instructions in the How to store raw data page.

TODO: [ANDREA] in the how to store raw data page, create a folders tree that includes all the relevant folders and subfolder. The current tree is not complete.

Anonymize raw scanner data¶

Expected project structure¶

You need to make sure your DICOM / nifti file names do not contain subject identificative information, such as the subject's name. This is particularly relevant for our pipeline, because that's exactly what our scanner does. At the hospital, it is best to manually name your files when exporting them. You can also use this small utility tool, which renames files within a BIDS-like directory structure, specifically targeting files containing _WIP_ in their names.

The script operates on the following project structure:

Project_Name/
├── sourcedata/
│   └── sub-xx/
│       ├── dicom/
│       ├── dicom_anon/
│       ├── bh/
│       ├── et/
│       └── nifti/
└── BIDS/
    ├── derivatives/
    └── sub-xx/
        ├── anat/
        └── func/

The script processes files within the 'sourcedata' directory.

Execute the script from the 'sourcedata' directory:

Open a terminal or command prompt.
Navigate to the project's root:

cd /path/to/Project_Name

Change to the 'sourcedata' directory:

cd sourcedata

Command-line Arguments¶

--level {group,participant}: Process all subjects (group) or individual subjects (participant).
--confirm {True,False}: Ask for confirmation before renaming (default: True).
--dry_run: Preview changes without renaming.
--sub [SUB ...]: Specify subject IDs to process.

Examples¶

Dry run for all subjects:

python /path/to/anon_nii_filename.py --level group --dry_run

Rename files for subjects 01 and 02 with confirmation:

python /path/to/anon_nii_filename.py --level participant --sub 01 02 --confirm True

Rename files for all subjects without confirmation:

python /path/to/anon_nii_filename.py --level group --confirm False

Output filename:

sub-01/nifti/sub-01_WIP_T1w_20240101141322.nii

Caution

Always backup your data before running renaming operations.

How It Works¶

The script traverses the 'sourcedata' directory structure.
It identifies files containing 'WIP' in their names.
New names are generated based on the subject ID and the part of the filename after 'WIP'.
Depending on the options, it either renames the files or shows the proposed changes.

Important Notes¶

Only files containing 'WIP' are processed. Others are ignored.
To process all files, modify the rename_files_in_directory function:

if '_WIP_' in file:

to:

if True:  # Caution: processes all files

Modifying the Script

Processing all files may lead to unintended renaming. Always use --dry_run first and review proposed changes carefully.

Tips¶

Use --dry_run to preview changes before actual renaming.
Process subjects in smaller batches for large datasets.
Regularly check BIDS specifications for naming convention compliance.

Troubleshooting¶

If issues occur:

Ensure you're in the 'sourcedata' directory.
Check permissions for renaming files in 'sourcedata'.
Verify all required Python dependencies are installed.
For unprocessed files, check if they contain 'WIP'.

Creating Event Files¶

Event files are crucial for analyzing fMRI data. They contain information about the timing and nature of stimuli or tasks during the scan. To create your event files manually:

Navigate to your sourcedata/sub-xx/bh/ folder.
Locate the behavioral output files (.mat or .log) for each run.
Create a corresponding events.tsv file for each run in the BIDS/sub-xx/func/ folder.

Each events.tsv file should contain at least three columns: onset, duration, and trial_type. Additional columns can be included as needed for your specific analysis.

If you use the fMRI task template, the log files you get as output contain all the information needed to build event files in a few steps. Below is a quick overview of the steps to take to make event files from log files. Note that it might not apply perfectly to all cases, and that other approaches can be more practical to you. It can be a good idea to create your own utility script to create event files from your behavioural results.

To create event files from log files, here is what you need to do (see the example below for an example transformation):

Create the onset column from the onset values in the log files: in the latter, onset times (usually stored in a column called ACTUAL_ONSET) are aligned to the start of the run. In BIDS event files, events need to be aligned to the start of the scanning. To obtain correct onset values, one can simply shift the onset of each line from a log file so that the onset 0.0 corresponds to the first TR trigger.
Create the duration column from the onset values. Log files typically don't record the exact duration of events, as that would put some extra calculation load onto MatLab (which struggles enough already as it is). A good approach is to calculate these post-hoc from the onset values, by simply taking the difference in between successive event onsets.
Create the trial_type column with the condition names. Fill this column by extracting the information that is relevant for your experimental design. In the example below, we extract the conditio names face and building from the EVENT_ID column, as these are the conditions we're interested in. We also indicate fixation where relevant, as we want to be able to model fixations in our GLM.
Add or keep any column you might need. In the example below, we keep the event_type, event_name and event_id columns as it might still be useful later on in the pipeline. Note that you should make a reference to these extra column in your events.json file.

Example log fileExample event file

EVENT_TYPE   EVENT_NAME  DATETIME                EXP_ONSET     ACTUAL_ONSET  DELTA       EVENT_ID
START        -           yyyy-mm-dd-hh-mm-ss     -               0.000000    -           -
FLIP         Instr       yyyy-mm-dd-hh-mm-ss     -               0.099950    -           -
RESP         KeyPress    yyyy-mm-dd-hh-mm-ss     -               7.663277    -           7
FLIP         TgrWait     yyyy-mm-dd-hh-mm-ss     -               7.697805    -           -
PULSE        Trigger     yyyy-mm-dd-hh-mm-ss     -              12.483778    -           5
PULSE        Trigger     yyyy-mm-dd-hh-mm-ss     -              24.452093    -           5
FLIP         Pre-fix     yyyy-mm-dd-hh-mm-ss     -              24.462263    -           -
PULSE        Trigger     yyyy-mm-dd-hh-mm-ss     -              26.452395    -           5
PULSE        Trigger     yyyy-mm-dd-hh-mm-ss     -              28.452362    -           5
PULSE        Trigger     yyyy-mm-dd-hh-mm-ss     -              30.451807    -           5
PULSE        Trigger     yyyy-mm-dd-hh-mm-ss     -              32.451339    -           5
PULSE        Trigger     yyyy-mm-dd-hh-mm-ss     -              34.451376    -           5
FLIP         Stim        yyyy-mm-dd-hh-mm-ss     34.462263      34.474302    0.012039    building_image.png
RESP         KeyPress    yyyy-mm-dd-hh-mm-ss     -              35.566808    -           9
FLIP         Fix         yyyy-mm-dd-hh-mm-ss     -              34.521628    -           -
PULSE        Trigger     yyyy-mm-dd-hh-mm-ss     -              36.451615    -           5
FLIP         Stim        yyyy-mm-dd-hh-mm-ss     37.462263      37.524439    0.062177    face_image.png
FLIP         Fix         yyyy-mm-dd-hh-mm-ss     -              37.572648    -           -
PULSE        Trigger     yyyy-mm-dd-hh-mm-ss     -              38.453535    -           5
RESP         KeyPress    yyyy-mm-dd-hh-mm-ss     -              38.806193    -           1
PULSE        Trigger     yyyy-mm-dd-hh-mm-ss     -              40.451415    -           5
...

onset            duration       event_type   event_name  event_id               trial_type
-24.452093       n/a            START        -           -                      n/a
-24.352143       7.597855       FLIP         Instr       -                      n/a
-16.788816       n/a            RESP         KeyPress    7                      n/a
-16.75428800     7.697805       FLIP         TgrWait     -                      n/a
-11.96831500     n/a            PULSE        Trigger     5                      n/a
0.0              n/a            PULSE        Trigger     5                      n/a
0.0101699999     10             FLIP         Pre-fix     -                      fixation
2.0003019999     n/a            PULSE        Trigger     5                      n/a
4.0002689999     n/a            PULSE        Trigger     5                      n/a
5.9997139999     n/a            PULSE        Trigger     5                      n/a
7.9992459999     n/a            PULSE        Trigger     5                      n/a
9.9992830000     n/a            PULSE        Trigger     5                      n/a
10.022209        0.0473259      FLIP         Stim        building_image.png     building
10.069534999     3.0028110      FLIP         Fix         -                      fixation
11.114715        n/a            RESP         KeyPress    9                      n/a
11.999521999     n/a            PULSE        Trigger     5                      n/a
13.072346        0.0482089      FLIP         Stim        face_image.png         face
13.120555        2.9033829      FLIP         Fix         -                      fixation
14.001442        n/a            PULSE        Trigger     5                      n/a
14.354099999     n/a            RESP         KeyPress    1                      n/a
15.999321999     n/a            PULSE        Trigger     5                      n/a
...

TODO: [ANDREA] Add script for automatically converting behavioral data to BIDS-compliant event files.

Converting DICOM files (Optional)¶

If you have collected DICOM files from the scanner, you need to 1. anonymise, and 2. convert them so that you can use them properly. There are several tools available that can take care of this. The dicm2nii GitHub repository is one such example. To use it, clone the repository, open MatLab, and follow these steps:

Navigate to your sourcedata folder.
Add the cloned dicm2nii folder to your path.
Use the anonymize_dicm script to anonymize the DICOM files. The command will look something like this:
```
anonymize_dicm('sub-xx/dicom', 'sub-xx/dicom_anon', 'sub-xx')
```

Use the dicm2nii script to convert the anonymized DICOM files to NIfTI.

dicm2nii('sub-xx/dicom_anon', 'sub-xx/dicom_converted', 'nii.gz')

These commands will have populated the dicom_anon and dicom_converted folders (see the folder trees in BIDS Conversion Overview for an example). The content of the latter can in turn be used to create json side car files (see Creating JSON Sidecar Files).

TODO: [ANDREA] add info about 1) example dcm2nii call, 2) why dcm2nii rather than dcm2niix, 3) add additional info on DICOM (no enhanced, missing values, etc.). see also this where Chris Rorden explains some practical issues with Phillips DICOMS, particularly the section on missing info (which we should probably link somewhere), and this thread, which explains issues with the enhanced DICOMs.

TODO: [TIM] Give information on how to use the anonymization and DICOM to nifti scripts, and what the results should be like. Give links to the scripts.

Creating JSON Sidecar Files¶

Each nii file must have a sidecar JSON file. However, if your fMRI protocol did not change, all the important JSON fields are going to be the same across different scanning sessions, and therefore JSON files can be re-used across subjects. This will save you some time, since getting DICOM files from the scanner can be quite time-consuming.

If you collected DICOM files for your participant, make sure you anonymised and converted your DICOM files and go through the following steps:
1. Locate the JSON sidecar files in sourcedata/sub-xx/dicom_converted/.
2. Open each JSON file and Complete the PhaseEncodingDirection and SliceTiming fields (see Missing fields in JSON files for more information).
3. Copy-paste the updated JSON files to accompany each NIfTI file in the BIDS/sub-xx/func folder: each run should have its accompanying sub-xx_task-taskname_run-xx_bold.json sidecar file.
If you did not collect DICOM files for your participant, but collected DICOM files for a previous participant and your fMRI protocol did not change in the meantime:
1. Copy-paste all .json sidecar files from the BIDS/sub-xx/func folder of the participant you have DICOM files for, to the BIDS/sub-xx/func folder of new participant you only collect nifti files for. Rename each file with the correct sub value, ensuring there is one .json file per participant, per run, with the correct name.

TODO: [TIM] Explain how to get the two missing fields and why it's important. Link to the fmri-general section about it.

TODO: [TIM] Explain how to duplicate and rename the sidecar file.

TODO: [ANDREA] fill this out with more in depth info about the JSON etc.

Renaming and Moving NIfTI Files¶

Navigate to your sourcedata/sub-xx/dicom_converted/ folder.
Identify the functional and structural NIfTI files.
Rename the files following BIDS conventions:
- Functional: sub-<label>_task-<label>_run-<label>_bold.nii
- Structural: sub-<label>_T1w.nii
Move the renamed files to their respective folders in BIDS/sub-xx/:
- Functional files go to BIDS/sub-xx/func/
- Structural files go to BIDS/sub-xx/anat/

TODO: [ANDREA] is dicom converted and nii the same folder? from which folder should we get the final nifti files to move? this folder needs to be consistent across different workflows (e.g., dicom conversion or just nifti files)

TODO: [TIM] Give instructions on how to rename files, both functional and structural, including what happens in case of several scan sessions and the added ses label.

Rename and move automatically

A MATLAB script that can do this automatically can be found here. Remember to change the input and output folders, run names and subjects numbers at the top of the script according to your needs. The script expects as input your nifti files, along with the JSON sidecar template files.

TODO: [ANDREA] the script should first check whether the JSON files are already availabe in the folder and, if that's the case, it should choose these files over the templates. Also, we need to add more info about these template files in this page!

Creating Essential BIDS Files¶

Create the following modality agnostic BIDS files files in your BIDS/ root folder:
- dataset_description.json
- participants.tsv
- participants.json
- task-<taskname>_bold.json
Fill in the required information for each file according to the BIDS specification.

Modality agnostic templates

In the modality agnostic BIDS files page, you can find templates and examples.

Setting Up Additional Components¶

Create a derivatives/ folder in your BIDS/ directory.
If needed, create a .bidsignore file in your BIDS/ root folder to exclude any non-BIDS compliant files.

Why should I use a .bidsignore file?

A `.bidsignore' file is useful to communicate to the BIDS validator which files should not be indexed, because they are not part of the standard BIDS structure. More information can be found here.

Validating Your BIDS Structure¶

Use the online BIDS Validator to check your BIDS structure.
Upload your entire BIDS/ folder and review any errors or warnings.
Make necessary corrections based on the validator's output.

By following these detailed steps, you'll ensure your data is properly organized in BIDS format, facilitating easier analysis and collaboration.

Now that you have your data in BIDS format, we can proceed to data pre-processing and quality assessment. See the next guide for instructions. → Pre-processing and QA