5ttgen error in qsiprep singularity container on HPC

1

Dear MRtrix-Community,

I am trying to deploy a qsiprep singularity container on our local CentOS HPC. Currently, I am stuck with an error regarding 5ttgen that appears to be challenging for me, the HPC admins and the qsiprep developers.

I run

singularity run --cleanenv --userns -B $proj_dir:/home/qsiprep\
    --home /home/qsiprep\
    -B $env_dir:/mnt \
    -B $bids_dir:/sngl/data \
    -B $deriv_dir:/sngl/out \
    -B $deriv_dir/qsiprep:/sngl/qsiprep-output \
    -B $scratch_dir:/sngl/scratch \
    -B $env_dir:/sngl/env \
    $env_dir/qsiprep-0.13.0RC2 \
    /sngl/data /sngl/qsiprep-output participant \
    -w /sngl/scratch/ \
    --participant-label ${subjs_subarr[@]} \
    --recon_input /sngl/out/qsiprep \
    --recon_spec mrtrix_singleshell_ss3t \
    --nthreads 16 \
    --fs-license-file /sngl/env/freesurfer_license.txt \
    --output-resolution 1.3

with $proj_dir, $env_dir, $bids_dir, $deriv_dir representing directories on a /work partition and $scratch_dir on a separate /scratch partition.

At some point in time 5ttgen is invoked

5ttgen fsl /sngl/out/qsiprep/sub-control01/anat/sub-control01_desc-preproc_T1w.nii.gz sub-control01_desc-preproc_T1w_5tt.mif -mask /sngl/out/qsiprep/sub-control01/anat/sub-control01_desc-brain_mask.nii.g

resulting in

5ttgen:
5ttgen: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-help option) for more information.
5ttgen:
5ttgen: Generated temporary directory: /sngl/scratch/qsirecon_wf/sub-control01_mrtrix_singleshell_ss3t/anat_ingress_wf/create_5tt/5ttgen-tmp-156GT5/
Command: mrconvert /sngl/out/qsiprep/sub-control01/anat/sub-control01_desc-preproc_T1w.nii.gz /sngl/scratch/qsirecon_wf/sub-control01_mrtrix_singleshell_ss3t/anat_ingress_wf/create_5tt/5ttgen-tmp-156GT5/input.mif
Command: mrconvert /sngl/out/qsiprep/sub-control01/anat/sub-control01_desc-brain_mask.nii.gz /sngl/scratch/qsirecon_wf/sub-control01_mrtrix_singleshell_ss3t/anat_ingress_wf/create_5tt/5ttgen-tmp-156GT5/mask.mif -datatype bit -strides -1,+2,+3
5ttgen: Changing to temporary directory (/sngl/scratch/qsirecon_wf/sub-control01_mrtrix_singleshell_ss3t/anat_ingress_wf/create_5tt/5ttgen-tmp-156GT5/)
Command: mrconvert input.mif T1.nii -strides -1,+2,+3
Command: mrcalc T1.nii mask.mif -mult T1_masked.nii.gz
Command: fast T1_masked.nii.gz
Command: run_first_all -m none -s L_Accu,R_Accu,L_Caud,R_Caud,L_Pall,R_Pall,L_Puta,R_Puta,L_Thal,R_Thal -i T1.nii -o first
5ttgen: Generating partial volume images for SGM structures… [=======================================================]
Command: mrmath mesh2voxel_L_Accu.mif mesh2voxel_R_Accu.mif mesh2voxel_L_Caud.mif mesh2voxel_R_Caud.mif mesh2voxel_L_Pall.mif mesh2voxel_R_Pall.mif mesh2voxel_L_Puta.mif mesh2voxel_R_Puta.mif mesh2voxel_L_Thal.mif mesh2voxel_R_Thal.mif sum - | mrcalc - 1.0 -min all_sgms.mif
5ttgen:
5ttgen: [ERROR] Command failed: mrmath mesh2voxel_L_Accu.mif mesh2voxel_R_Accu.mif mesh2voxel_L_Caud.mif mesh2voxel_R_Caud.mif mesh2voxel_L_Pall.mif mesh2voxel_R_Pall.mif mesh2voxel_L_Puta.mif mesh2voxel_R_Puta.mif mesh2voxel_L_Thal.mif mesh2voxel_R_Thal.mif sum - | mrcalc - 1.0 -min all_sgms.mif (fsl.py:175)
5ttgen: Output of failed command:
mrmath: computing sum across 10 images… [==========]
mrmath: [SYSTEM FATAL CODE: SIGBUS (7)] Bus error: Accessing invalid address (out of storage space?)
mrcalc: [ERROR] Could not interpret string “-” as either an image path or a numerical value
5ttgen:
5ttgen: Changing back to original directory (/sngl/scratch/qsirecon_wf/sub-control01_mrtrix_singleshell_ss3t/anat_ingress_wf/create_5tt)
5ttgen: Script failed while executing the command: mrmath mesh2voxel_L_Accu.mif mesh2voxel_R_Accu.mif mesh2voxel_L_Caud.mif mesh2voxel_R_Caud.mif mesh2voxel_L_Pall.mif mesh2voxel_R_Pall.mif mesh2voxel_L_Puta.mif mesh2voxel_R_Puta.mif mesh2voxel_L_Thal.mif mesh2voxel_R_Thal.mif sum - | mrcalc - 1.0 -min all_sgms.mif
5ttgen: For debugging, inspect contents of temporary directory: /sngl/scratch/qsirecon_wf/sub-control01_mrtrix_singleshell_ss3t/anat_ingress_wf/create_5tt/5ttgen-tmp-156GT5/
Return code: 1

Apparently, mrmath results in a memory access issue. As far as I understand, mrmath should read from and write to /scratch. According to the admins /scratch has plenty of storage space left (only 12gb of 90gb occupied). Interestingly, if I run 5ttgen outside of the singularity container with appropriately adapted paths everything works fine. Correspondingly, the exact mrmath command (in the temporary directory created by 5ttgen) doesn’t work as well.

For the sake of completeness:
HPC singularity version - 3.5.2
qsiprep singularity container - 0.13.0RC2

I have the hope to get some insights about what mrmath is doing so that the admins and me could further address the issue.

Thanks in advance.

2

I’m not sure how qsiprep works exactly, this might be something you can discuss with the maintainers of that project. Nonetheless, I can certainly offer my 2¢ on the matter…

The relevant part of the output is this (as you’re already identified):

And the most likely reason is indeed that hinted at by the error message: out of space. The reason there’s confusion here is that the storage space used at this stage is likely not the scratch folder…

This is a Unix pipe, with one command (mrmath) feeding its output to the next command (mrcalc). The way this is implemented in MRtrix is outlined in the docs, but briefly: the first command writes the data to a temporary file, feeds the filename of that file to the next command, and the next command reads from that file (and deletes it afterwards). The location of that temporary file is set to /tmp by default (mostly to ensure it remains local to the compute node it’s running on to avoid needless round trips over the network). This can be modified via the configuration file, or via an environment variable (which take precedence over the config file).

In short: if there isn’t much space in the container’s /tmp folder, or for some reason it’s already being used to capacity, then you’ll get this error. I’m not sure whether the maintainers of qsiprep have altered the default location for these files, but this is the first place I would start digging…

3

Hi Donald,
thanks a lot for your immediate response. Pointing towards /tmp led to the solution. Only 25mb are assigned to /tmp on our HPC. Setting MRTRIX_TMPFILE_DIR=/scratch solved the issue.

1 Like