MRtrix Singularity container: eddy_cuda fails to run

1

Dear experts,

I am using MRtrix3 Singularity container to run dwifslpreproc and eddy_cuda was unsuccessful. I’ve seen many threads relating to this issue but couldn’t find a solution for the problem. Your help would be much appreciated!!

Singularity container image: docker://mrtrix3/mrtrix3:latest
Nvidia GPU: Available, through job submission (Slurm) to a central processing cluster
Cuda versions supported: From 9.0 to 12.3
Eddy command called during dwifslpreproc: eddy_cuda10.2 --imain=eddy_in.nii --mask=eddy_mask.nii --acqp=eddy_config.txt --index=eddy_in$
Result: dwifslpreproc: CUDA version of ‘eddy’ was not successful; attempting OpenMP version

Remarks:

  1. Running other MRtrix3 functions (e.g., mrconvert, mrcat, etc.) are completely fine.
  2. Running “eddy_cuda” with FSL Singularity container is also fine (but it calls eddy_cuda9.1 instead of eddy_cuda10.2 as in the case of dwifslpreproc)
  3. I don’t have FSL installed locally. Is this relevant?

I’m a beginner of using Singularity. Sorry if any of these sounds stupid!! Most grateful if you could find out the cause of the problem. It’s okay if it is unsolvable - just want to have an idea of what’s wrong.

Thank you so much!!!

2

Hi. Not sure about docker, but in singularity I believe the option --nv is necessary for the container to use the GPU. You could convert your docker container to singularity/apptainer and see if that works… Good luck!

3

Hey - some troubleshooting thoughts:

  • As the other user remarked, --nv is necessary in your Singularity call to have GPU passthrough assuming your job has landed onto a GPU compute node
  • You should check with your cluster team that the CUDA version called by dwifslpreproc (i.e., CUDA 10.2) is actually available and configured properly. It sounds like a mismatch issue if you can use the eddy_cuda from the FSL Singularity container. You can probably check this by using an interactive session on the GPU node and calling nvidia-smi to check.
  • You don’t need FSL to be installed locally! The MRtrix3 container wraps into it the relevant FSL functions.

Hope that helps!