multi-shell-dMRIharmonization repository is developed by Tashrif Billah and Yogesh Rathi, Brigham and Women's Hospital (Harvard Medical School).
multi-shell-dMRIharmonization is an extension of dMRIharmonization for single-shell dMRI.
- Algorithm
- Citation
- Dependencies
- Installation
- Running
- Consistency checks
- Varying number of gradients
- Sample commands
- Tests
- Preprocessing
- Debugging
- Caveats/Issues
- Reference
Table of Contents created by gh-md-toc
- Extract b-shells from given data
- Check consistency among bshells and spatial resolution
- Create ANTs template from highest b-shell
- Apply the warps and affines obtained from previous step to compute scale maps for all b-shells
- Template creation being complete, harmonize data for each b-shell using scale maps corresponding to that b-shell
- Join the harmonized data in the same order of bvalues as that of the given data
Template creation and data harmonization process mentioned in steps 3-5 are described in detail at single shell counterpart of the program:
https://github.com/pnlbwh/dMRIharmonization#template-creation
https://github.com/pnlbwh/dMRIharmonization#data-harmonization
If this repository is useful in your research, please cite all of the following:
-
Billah T, Bouix S, Rathi Y. Multi-site multi-shell Diffusion MRI Harmonization, https://github.com/pnlbwh/multi-shell-dMRIharmonization, 2019, doi: 10.5281/zenodo.3451427
-
Billah T*, Cetin Karayumak S*, Bouix S, Rathi Y. Multi-site Diffusion MRI Harmonization, https://github.com/pnlbwh/dMRIharmonization, 2019, doi: 10.5281/zenodo.2584275
* denotes equal first authroship
-
Cetin Karayumak S, Bouix S, Ning L, James A, Crow T, Shenton M, Kubicki M, Rathi Y. Retrospective harmonization of multi-site diffusion MRI data acquired with different acquisition parameters. Neuroimage. 2019 Jan 1;184:180-200. doi: 10.1016/j.neuroimage.2018.08.073. Epub 2018 Sep 8. PubMed PMID: 30205206; PubMed Central PMCID: PMC6230479.
-
Mirzaalian H, Ning L, Savadjiev P, Pasternak O, Bouix S, Michailovich O, Karmacharya S, Grant G, Marx CE, Morey RA, Flashman LA, George MS, McAllister TW, Andaluz N, Shutter L, Coimbra R, Zafonte RD, Coleman MJ, Kubicki M, Westin CF, Stein MB, Shenton ME, Rathi Y. Multi-site harmonization of diffusion MRI data in a registration framework. Brain Imaging Behav. 2018 Feb;12(1):284-295. doi:10.1007/s11682-016-9670-y. PubMed PMID: 28176263.
-
Two groups of data from- reference and target sites are required. Control (healthy) subjects should be present in each site.
-
The groups between the sites should be very well matched for age, sex, socio-economic status, IQ and any other demographic variable.
-
A minimum of 16 subjects is required from each site for proper harmonization (so a minimum of 32 subjects in total).
-
The data should be curated with the following steps prior to harmonization:
(i) axis alignment and centering
(ii) signal dropped gradient removal
(iii) eddy current and head motion correction
-
b-values in each b-shell should have similar b-values (i.e, if one site has b-value 1000, the other one should have in the range [900,1100]).
If your data does not satisfy these requirements, please open an issue here or contact -
- ANTs = 2.2.0
- reisert/unring
- FSL = 5.0.11
- numpy = 1.16.2
- scipy = 1.2.1
- scikit-image = 0.15.0
- dipy = 0.16.0
- nibabel = 2.3.0
- pynrrd = 0.3.6
- conversion = 2.0
NOTE The above versions were used for developing the repository. However, multi-shell-dMRIharmonization should work on any advanced version.
You may ignore installation instruction for any software module that you have already.
uname -a # check if 32 or 64 bit
Download Miniconda Python 3.6 bash installer (32/64-bit based on your environment):
sh Miniconda3-latest-Linux-x86_64.sh -b # -b flag is for license agreement
Activate the conda environment:
source ~/miniconda3/bin/activate # should introduce '(base)' in front of each line
NOTE With the current design, MATLAB Runtime Compiler and unringing are not used. So, you may pass them.
In the harmonization process, all volumes have to be resampled to a common spatial resolution. We have chosen bspline as the interpolation method. For better result, bspline order has been chosen to be 7. Since Python and ANTs provide bspline order less than or equal to 5, we have resorted to spm package for this task. Using their source codes bspline.c and bsplins.c, we have made a standalone executable that performs the above interpolation. Execution of the standalone executable requires MATLAB Runtime Compiler. It is available free of charge. Download a suitable version from the above, and install as follows:
unzip MCR_R2017a_glnxa64_installer.zip -d MCR_R2017a_glnxa64/
MCR_R2017a_glnxa64/install -mode silent -agreeToLicense yes -destinationFolder `pwd`/MATLAB_Runtime
See details about installation here.
After successful installation, you should see a suggestion about editing your LD_LIBRARY_PATH.
You should save the suggestion in a file env.sh.
echo "/path/to/v92/runtime/glnxa64:/path/to/v92/bin/glnxa64:/path/to/v92/sys/os/glnxa64:/path/to/v92/opengl/lib/glnxa64:" > env.sh
Then, every time you run dMRIharmonization, you can just source the env.sh for your LD_LIBRARY_PATH to be updated.
NOTE If you have MATLAB already installed in your system, replace /path/to/v92 with /path/to/Matlab/
git clone https://bitbucket.org/reisert/unring.git
cd unring/fsl
export PATH=$PATH:`pwd`
You should be able to see the help message now:
unring.a64 --help
NOTE FSL unringing executable requires Centos7 operating system.
Now that you have installed the prerequisite software, you are ready to install the pipeline:
git clone https://github.com/pnlbwh/multi-shell-dMRIharmonization.git && cd multi-shell-dMRIharmonization
conda env create -f environment.yml # you may comment out any existing package from environment.yml
conda activate harmonization # should introduce '(harmonization)' in front of each line
Alternatively, if you already have ANTs, you can continue using your python environment by directly installing the prerequisite libraries:
pip install -r requirements.txt --upgrade
dMRIharmonization toolbox is provided with a debugging capability to test how good has been the
harmonization. For debug to work and tests to run, download the following data from IIT HUMAN BRAIN ATLAS
and place them in multi-shell-dMRIharmonization/IITAtlas/ directory:
Make sure the following executables are in your path:
antsMultivariateTemplateConstruction2.sh
antsApplyTransforms
antsRegistrationSyNQuick.sh
unring.a64
You can check them as follows:
which dtifit
If any of them does not exist, add that to your path:
export PATH=$PATH:/directory/of/executable
conda activate harmonization should already put the ANTs scripts in your path. Yet, you should set ANTSPATH as follows:
export ANTSPATH=~/miniconda3/envs/harmonization/bin/
However, if you choose to use pre-installed ANTs scripts, you can define ANTSPATH according to this instruction.
Upon successful installation, you should be able to see the help message:
lib/multi-shell-harmonization.py --help
Usage:
multi-shell-harmonization.py [SWITCHES]
Meta-switches:
-h, --help Prints this help message and quits
--help-all Prints help messages of all sub-commands and quits
-v, --version Prints the program's version and quits
Switches:
--create turn on this flag to create template
--debug turn on this flag to debug harmonized data (valid only with --process)
--force turn on this flag to overwrite existing data
--nproc VALUE:str number of processes/threads to use (-1 for all available, may slow down your system);
the default is 4
--nshm VALUE:str spherical harmonic order, by default maximum possible is used; the default is -1
--nzero VALUE:str number of zero padding for denoising skull region during signal reconstruction; the
default is 10
--process turn on this flag to harmonize
--ref_list VALUE:ExistingFile reference csv/txt file with first column for dwi and 2nd column for mask:
dwi1,mask1\ndwi2,mask2\n...
--ref_name VALUE:str reference site name; required
--tar_list VALUE:ExistingFile target csv/txt file with first column for dwi and 2nd column for mask:
dwi1,mask1\ndwi2,mask2\n...
--tar_name VALUE:str target site name; required
--template VALUE:str template directory; required
--travelHeads travelling heads
--verbose print everything to STDOUT
For details about the above arguments, see https://github.com/pnlbwh/dMRIharmonization#running
A few consistency checks are run to make sure provided data is eligible for harmonization.
-
First image from the reference site is used as reference for all images in reference and target sites. B-shells and spatial resolution are extracted from the reference image. The b-shells and spatial resolution are compared against all images in reference and target sites. As long as all bvalues of each image falls within +-100 of a b-shell bvalue, the image is considered good for harmonization.
-
Spatial resolution should match exactly. If reference image has 2x2x2 resolution, all other images should also have this resolution.
-
The program can automatically determine maximum possible spherical harmonic order (
--nshm) for each b-shell. However, if a value is provided with--nshm, it is compared against the maximum possible spherical harmonic order to continue.
In the event of inconsistency, the program will raise and error and user should remove the case from provided lists and try again.
A particular strength of the algorithm is its compatiblity with varying number of gradients present in images.
As long as the number of gradients satisfies minimum required for the spherical harmonic order, determined (--nshm -1)
or provided(--nshm 4), data can be harmonized.
multi-shell-dMRIharmonization/lib/multi-shell-harmonization.py --ref_list ref_list.txt --tar_list tar_list.txt
--ref_name REF --tar_name TAR --template template
--create
multi-shell-dMRIharmonization/lib/multi-shell-harmonization.py --tar_list tar_list.txt
--tar_name TAR --template template
--process
multi-shell-dMRIharmonization/lib/multi-shell-harmonization.py --tar_list tar_list.txt
--tar_name TAR --template template
--process --debug
Run together with --create and --process:
multi-shell-dMRIharmonization/lib/multi-shell-harmonization.py --ref_list ref_list.txt --tar_list tar_list.txt
--ref_name REF --tar_name TAR --template template
--create --process --debug
In theory, you want to create a template with small number of data from each sites and then use the template to
harmonize all of your data in the target site. Since the data for template creation and harmonization are not same,
we don't have luxury of using --create --process --debug altogether. Instead, you would use --debug with each of
--create and --process. The --debug flag creates some files that are used to obtain statistics later.
The steps are described below:
(i) Create template with --debug enabled:
multi-shell-dMRIharmonization/lib/multi-shell-harmonization.py --ref_list ref_list.txt --tar_list tar_small_list.txt
--ref_name REF --tar_name TAR --template template
--create --debug
(ii) Harmonize data with --debug enabled:
multi-shell-dMRIharmonization/lib/multi-shell-harmonization.py --tar_list tar_list.txt
--tar_name TAR --template template
--process --debug
(iii) Obtain statistics
## reference site ##
# start with highest bvalue shell
multi-shell-dMRIharmonization/lib/tests/fa_skeleton_test.py
-i ref_list_b3000.csv.modified -s REF
-t template/ --bshell_b 3000
# repeat for other non-zero bvalues
multi-shell-dMRIharmonization/lib/tests/fa_skeleton_test.py
-i ref_list_b2000.csv.modified -s REF
-t template/ --bshell_b 2000
multi-shell-dMRIharmonization/lib/tests/fa_skeleton_test.py
-i ref_list_b1000.csv.modified -s REF
-t template/ --bshell_b 1000
...
...
## target site before harmonization ##
# again, start with highest bvalue shell, notice the absence of ".modified" at the end of -i
multi-shell-dMRIharmonization/lib/tests/fa_skeleton_test.py
-i tar_list_b3000.csv -s TAR
-t template/ --bshell_b 3000
# repeat for other non-zero bvalues
...
...
## target site after harmonization ##
# once again, start with highest bvalue shell, notice the presence of ".modified.harmonized" at the end of -i
multi-shell-dMRIharmonization/lib/tests/fa_skeleton_test.py
-i tar_list_b3000.csv.modified.harmonized -s TAR
-t template/ --bshell_b 3000
# repeat for other non-zero bvalues
...
...
A small test data is provided with each release. TBD Tests are incomplete as of now but will be completed soon.
You may test the whole pipeline as follows:
cd multi-shell-dMRIharmonization/lib/tests
./multi_pipeline_test.sh
NOTE Running the above tests might take six hours.
* If there is any problem downloading test data, try manually downloading and unzipping it to lib/tests/ folder.
You may run smaller and faster unittest as follows:
python -m unittest discover -v lib/tests/
Unlike single-shell dMRIharmonization, multi-shell-dMRIharmonization does NOT support data preprocessing as of now. This is likely to change in a future release. Hence, the following arguments are present for legacy purpose but you should not use them.
--denoise # turn on this flag to denoise voxel data
--bvalMap VALUE # specify a bmax to scale bvalues into
--resample VALUE # voxel size MxNxO to resample into
After preprocessing, the image lists are saved with .modified extension in the same location of provided lists,
and used for further processing.
multi-shell-dMRIharmonization debugging is an extension of dMRIharmonization debugging. In the former case, debugging is run at each shell separately. So, at each shell you should see nearly matching mean FA over IITmean_FA_skeleton.nii.gz between reference site and target site after harmonization:
REF mean FA: 0.5217237675408243
TAR mean FA before harmonization: 0.5072286796848892
REF mean FA after harmonization: 0.5321998242139347
--debug should run together with --create and --process since information from reference and target sites are used
to obtain the above summary.
However, if you would like to debug separately or if your target site has more cases than the ones used in template creation,
we provide you a way to debug manually. Firstly, make sure to harmonize data with --debug flag enabled. The flag will
create diffusion measures that are used in debugging later. Then, lib/test/fa_skeleton_test.py script can register
each subject FA (reference, target before harmonization, and after harmonization), to template space and then to MNI space.
usage: fa_skeleton_test.py [-h] -i INPUT -s SITE -t TEMPLATE --bshell_b
BSHELL_B [--ncpu NCPU]
Warps diffusion measures (FA, MD, GFA) to template space and then to MNI
space. Finally, calculates mean FA over IITmean_FA_skeleton.nii.gz
optional arguments:
-h, --help show this help message and exit
-i INPUT, --input INPUT
a .txt/.csv file that you used in/obtained from
harmonization.py having two columns for (img,mask)
pair. See pnlbwh/dMRIharmonization documentation for
more details
-s SITE, --site SITE site name for locating template FA and mask in
tempalte directory
-t TEMPLATE, --template TEMPLATE
template directory where Mean_{site}_FA.nii.gz and
{site}_Mask.nii.gz is located
--bshell_b BSHELL_B bvalue of the bshell
--ncpu NCPU number of cpus to use
Finally, it should print mean FA statistics like above.
lib/test/fa_skeleton_test.py performs two registrations:
(i) Subject to site template space
(ii) Site template to MNI space
Thus, subject data is brought to MNI space for comparison. For reference site, it is a straightforward process. However, for target site, there are two kinds of data: given and harmonized. Since given data and harmonized data reside in the same space, registration is performed only once. The script is intelligent enough to exploit relevant registration files if registration was performed before.
In multi-shell-dMRIharmonization approach, registration is performed with highest b-shell. Obtained transform files are used to warp rest of the b-shells. See Different target list for details.
antsMultivariateTemplateConstruction2.sh: all the images need to have unique
prefix because transform files are created in the same --template ./template/ directory. The value of --template
should have / at the end. The pipeline appends one if there is not, but it is good to include it when specifying.
Multi threading requires memory and processor availability. If pipeline does not continue past
unringing or shm_coeff computation, your machine likely ran out of memory. Reducing --nproc to lower number of processors (i.e. 1-4)
or swithcing to a powerful machine should help in this regard.
Standalone MATLAB executable bspline used for resampling in the pipeline, requires X forwarding to be properly set.
If it is not properly set, you may notice error like below:
X11 proxy: unable to connect to forwarded X server: Network error: Connection refused
Either of the following should fix that:
ssh -X user@remotehost
or
ssh user@remotehost
unset DISPLAY
When using the latter option, be mindful that it may cause other programs requiring $DISPLAY
in that particular terminal to malfunction.
In any case, feel free to submit an issue here. We shall get back to you as soon as possible.
Zhang S, Arfanakis K. Evaluation of standardized and study-specific diffusion tensor imaging templates of the adult human brain: Template characteristics, spatial normalization accuracy, and detection of small inter-group FA differences. Neuroimage 2018;172:40-50.
Billah, Tashrif; Bouix, Sylvain; Rathi, Yogesh; Various MRI Conversion Tools, https://github.com/pnlbwh/conversion, 2019, DOI: 10.5281/zenodo.2584003.