Dear MRtrix Community,
I am currently working on a qsiprep implementation with the datalad fairly big processing workflow. I use the containerized qsiprep version 0.14.2 from dockerhub.
I face an error with dwi2response that I don’t exactly know how to diagnose and address.
QSIRecon failed: Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 428, in run
runtime = self._run_interface(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 822, in _run_interface
self.raise_exception(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 753, in raise_exception
).format(**runtime.dictcopy())
RuntimeError: Command:
dwi2response dhollander -mask /dev/shm/1332847.10143549/ds/qsirecon_wf/sub-1332847_mrtrix_multishell_msmt/recon_wf/msmt_csd/_dwi_file_..dev..shm..1332847.10143549..ds..outputs..qsiprep..sub-1332847..ses-2..dwi..sub-1332847_ses-2_space-T1w_desc-preproc_dwi.nii.gz/resample_mask/sub-1332847_desc-brain_mask_resample.nii.gz -nthreads 16 /dev/shm/1332847.10143549/ds/qsirecon_wf/sub-1332847_mrtrix_multishell_msmt/recon_wf/msmt_csd/_dwi_file_..dev..shm..1332847.10143549..ds..outputs..qsiprep..sub-1332847..ses-2..dwi..sub-1332847_ses-2_space-T1w_desc-preproc_dwi.nii.gz/create_mif/sub-1332847_ses-2_space-T1w_desc-preproc_dwi.mif sub-1332847_ses-2_space-T1w_desc-preproc_dwi_wm.txt sub-1332847_ses-2_space-T1w_desc-preproc_dwi_gm.txt sub-1332847_ses-2_space-T1w_desc-preproc_dwi_csf.txt
Standard output:
Standard error:
mrstats: [ERROR] Cannot output statistic of interest; no values read (empty mask?)
dwi2response:
dwi2response: Note that this script makes use of commands / algorithms that have relevant articles for citation. Please consult the help page (-help option) for more information.
dwi2response:
dwi2response: Generated scratch directory: /dev/shm/1332847.10143549/ds/qsirecon_wf/sub-1332847_mrtrix_multishell_msmt/recon_wf/msmt_csd/_dwi_file_..dev..shm..1332847.10143549..ds..outputs..qsiprep..sub-1332847..ses-2..dwi..sub-1332847_ses-2_space-T1w_desc-preproc_dwi.nii.gz/estimate_response/dwi2response-tmp-BW9OCD/
dwi2response: Importing DWI data (/dev/shm/1332847.10143549/ds/qsirecon_wf/sub-1332847_mrtrix_multishell_msmt/recon_wf/msmt_csd/_dwi_file_..dev..shm..1332847.10143549..ds..outputs..qsiprep..sub-1332847..ses-2..dwi..sub-1332847_ses-2_space-T1w_desc-preproc_dwi.nii.gz/create_mif/sub-1332847_ses-2_space-T1w_desc-preproc_dwi.mif)...
dwi2response: Importing mask (/dev/shm/1332847.10143549/ds/qsirecon_wf/sub-1332847_mrtrix_multishell_msmt/recon_wf/msmt_csd/_dwi_file_..dev..shm..1332847.10143549..ds..outputs..qsiprep..sub-1332847..ses-2..dwi..sub-1332847_ses-2_space-T1w_desc-preproc_dwi.nii.gz/resample_mask/sub-1332847_desc-brain_mask_resample.nii.gz)...
dwi2response: Changing to scratch directory (/dev/shm/1332847.10143549/ds/qsirecon_wf/sub-1332847_mrtrix_multishell_msmt/recon_wf/msmt_csd/_dwi_file_..dev..shm..1332847.10143549..ds..outputs..qsiprep..sub-1332847..ses-2..dwi..sub-1332847_ses-2_space-T1w_desc-preproc_dwi.nii.gz/estimate_response/dwi2response-tmp-BW9OCD/)
dwi2response: -------
dwi2response: 2 unique b-value(s) detected: 5,2003 with 3,3 volumes
dwi2response: -------
dwi2response: Preparation:
dwi2response: * Eroding brain mask by 3 pass(es)...
dwi2response: [ mask: 189527 -> 144840 ]
dwi2response: * Computing signal decay metric (SDM):
dwi2response: * b=5...
dwi2response: * b=2003...
dwi2response: * Removing erroneous voxels from mask and correcting SDM...
dwi2response: [ mask: 144840 -> 144817 ]
dwi2response: -------
dwi2response: Crude segmentation:
dwi2response: * Crude WM versus GM-CSF separation (at FA=0.2)...
dwi2response: [ 144817 -> 144817 (WM) & 0 (GM-CSF) ]
dwi2response: * Crude GM versus CSF separation...
dwi2response: [ERROR] Error trying to calculate statistic 'median' from image 'safe_sdm.mif'
Return code: 1
Traceback (most recent call last):
File "/usr/local/miniconda/bin/qsiprep", line 8, in <module>
sys.exit(main())
File "/usr/local/miniconda/lib/python3.7/site-packages/qsiprep/cli/run.py", line 721, in main
qsirecon_post_wf.run(**plugin_settings)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
runner.run(execgraph, updatehash=updatehash, config=self.config)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/base.py", line 164, in run
self._clean_queue(jobid, graph, result=result)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/base.py", line 227, in _clean_queue
raise RuntimeError("".join(result["traceback"]))
RuntimeError: Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 428, in run
runtime = self._run_interface(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 822, in _run_interface
self.raise_exception(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 753, in raise_exception
).format(**runtime.dictcopy())
RuntimeError: Command:
dwi2response dhollander -mask /dev/shm/1332847.10143549/ds/qsirecon_wf/sub-1332847_mrtrix_multishell_msmt/recon_wf/msmt_csd/_dwi_file_..dev..shm..1332847.10143549..ds..outputs..qsiprep..sub-1332847..ses-2..dwi..sub-1332847_ses-2_space-T1w_desc-preproc_dwi.nii.gz/resample_mask/sub-1332847_desc-brain_mask_resample.nii.gz -nthreads 16 /dev/shm/1332847.10143549/ds/qsirecon_wf/sub-1332847_mrtrix_multishell_msmt/recon_wf/msmt_csd/_dwi_file_..dev..shm..1332847.10143549..ds..outputs..qsiprep..sub-1332847..ses-2..dwi..sub-1332847_ses-2_space-T1w_desc-preproc_dwi.nii.gz/create_mif/sub-1332847_ses-2_space-T1w_desc-preproc_dwi.mif sub-1332847_ses-2_space-T1w_desc-preproc_dwi_wm.txt sub-1332847_ses-2_space-T1w_desc-preproc_dwi_gm.txt sub-1332847_ses-2_space-T1w_desc-preproc_dwi_csf.txt
Standard output:
Standard error:
mrstats: [ERROR] Cannot output statistic of interest; no values read (empty mask?)
dwi2response:
dwi2response: Note that this script makes use of commands / algorithms that have relevant articles for citation. Please consult the help page (-help option) for more information.
dwi2response:
dwi2response: Generated scratch directory: /dev/shm/1332847.10143549/ds/qsirecon_wf/sub-1332847_mrtrix_multishell_msmt/recon_wf/msmt_csd/_dwi_file_..dev..shm..1332847.10143549..ds..outputs..qsiprep..sub-1332847..ses-2..dwi..sub-1332847_ses-2_space-T1w_desc-preproc_dwi.nii.gz/estimate_response/dwi2response-tmp-BW9OCD/
dwi2response: Importing DWI data (/dev/shm/1332847.10143549/ds/qsirecon_wf/sub-1332847_mrtrix_multishell_msmt/recon_wf/msmt_csd/_dwi_file_..dev..shm..1332847.10143549..ds..outputs..qsiprep..sub-1332847..ses-2..dwi..sub-1332847_ses-2_space-T1w_desc-preproc_dwi.nii.gz/create_mif/sub-1332847_ses-2_space-T1w_desc-preproc_dwi.mif)...
dwi2response: Importing mask (/dev/shm/1332847.10143549/ds/qsirecon_wf/sub-1332847_mrtrix_multishell_msmt/recon_wf/msmt_csd/_dwi_file_..dev..shm..1332847.10143549..ds..outputs..qsiprep..sub-1332847..ses-2..dwi..sub-1332847_ses-2_space-T1w_desc-preproc_dwi.nii.gz/resample_mask/sub-1332847_desc-brain_mask_resample.nii.gz)...
dwi2response: Changing to scratch directory (/dev/shm/1332847.10143549/ds/qsirecon_wf/sub-1332847_mrtrix_multishell_msmt/recon_wf/msmt_csd/_dwi_file_..dev..shm..1332847.10143549..ds..outputs..qsiprep..sub-1332847..ses-2..dwi..sub-1332847_ses-2_space-T1w_desc-preproc_dwi.nii.gz/estimate_response/dwi2response-tmp-BW9OCD/)
dwi2response: -------
dwi2response: 2 unique b-value(s) detected: 5,2003 with 3,3 volumes
dwi2response: -------
dwi2response: Preparation:
dwi2response: * Eroding brain mask by 3 pass(es)...
dwi2response: [ mask: 189527 -> 144840 ]
dwi2response: * Computing signal decay metric (SDM):
dwi2response: * b=5...
dwi2response: * b=2003...
dwi2response: * Removing erroneous voxels from mask and correcting SDM...
dwi2response: [ mask: 144840 -> 144817 ]
dwi2response: -------
dwi2response: Crude segmentation:
dwi2response: * Crude WM versus GM-CSF separation (at FA=0.2)...
dwi2response: [ 144817 -> 144817 (WM) & 0 (GM-CSF) ]
dwi2response: * Crude GM versus CSF separation...
dwi2response: [ERROR] Error trying to calculate statistic 'median' from image 'safe_sdm.mif'
Return code: 1
Apparently the mask isn’t empty but GM-CSF is. I get the same error for all 16 test subjects I run in parallel. Inputs are structural and diffusion data from the UK biobank.
Some notes to my approach. I use a slightly customized version from GitHub - psychoinformatics-de/fairly-big-processing-workflow: A template for decentralized, reproducible processing and I don’t find an obvious reason why dwi2response is failing while the remaining non-dependent commands succeed. Max RAM usage of the jobs is 256gb of 1 TB available. Suspecting an issue with /tmp I mounted the current directory /dev/shm/1332847.10143549/ds/ also as /tmp in the container and set it explicitly with SINGULARITYENV_TMPFILE_DIR=/tmp. If I run qsiprep without any datalad involvement on the same system it finishes without issues pointing me away from qsiprep/mrtrix and towards datalad / the fairly big workflow. However, to get an idea of what might be wrong / where to look I hope for some insights from your site. This has also been posted as a qsiprep github issue (dwi2response: [ERROR] Error trying to calculate statistic 'median' from image 'safe_sdm.mif' · Issue #337 · PennLINC/qsiprep · GitHub).
Happy to provide further details and grateful for any input.
Cheers,
Marvin