mrtrix on m3@massive.org.au eddy_cuda9.1

1

Hi Folks
I am trying to use mrtrix on massive, and I am trying to use the cuda version of eddy, specifically eddy_cuda9.1
It fails complaining about a lacking library libcudart.so.9.1
I tried adding the cuda module

module load cuda/9.1
but it still fails.

For those unfamiliar with the CVL system on M3 there a large number of containers with software packages available that can be loaded. For mrtrix there are:

[mschira@m3-login2]$ module load mrtrix
mrtrix/0.3.15-gcc4 mrtrix/0.3.16 mrtrix/3.0.0 mrtrix/3.0_rc3_latest mrtrix3tissue/5.2.8-ubuntu
mrtrix/0.3.15-gcc5 mrtrix/20170712 mrtrix/3.0_rc3 mrtrix3tissue/5.2.8

So far I have been using mrtrix/3.0.1

maybe I should try another module?
best
Mark

2

Hi Mark,

I personally have eddy_cuda9.1 working on M3, but within a Singularity container.

Upon loading the CUDA module, check the contents of your LD_LIBRARY_PATH environment variable; it should be:
/usr/local/cuda/9.1/extras/CUPTI/lib64:/usr/local/cuda/9.1/lib64:/usr/local/cuda/9.1/lib:/usr/local/cuda/9.1/lib64/stubs:/opt/munge-0.5.11/lib:/opt/slurm-19.05.4/lib:/opt/slurm-19.05.4/lib/slurm
Assuming that comes out as expected, that’s pretty much the extent of the feedback that I can offer, since knowledge of that envvar was the majority of what I needed to get it working within Singularity.

I would also note that after loading cuda/9.1 and fsl/6.0.3, I could read the eddy_cuda9.1 help page just fine, which would have failed if libcudart.so.9.1 weren’t found for me either:

[robertes@m3-login2 hb93]$ ldd `which eddy_cuda9.1` | grep libcudart
libcudart.so.9.1 => /usr/local/cuda/9.1/lib64/libcudart.so.9.1 (0x00007fe5b9082000)

So it might be something specific to your account?

The M3 help team have been awesome to me, so don’t be afraid to reach out to them.

Cheers
Rob

3

Hi Rob
Do you have a singularity container for mrtrix working for M3?
If so could you just share it with me?
Cheers
Mark
P.S. yes the massive team is typically very helpful, but I was just thinking there must be a bunch of guys using mrtrix on massive…

4

Mark

The container I myself have on M3 encapsulates all of the dependencies for a specific BIDS App analysis pipeline, and runs that pipeline as the entrypoint; so it’s not intended for general use of individual MRtrix3 functionalities.

I have been working on providing “official” MRtrix3 containers, and had hoped to have it ready to go out with 3.0.2, but there’s still some discrepancies to iron out. I’ll consider your question as an implicit “bump” to get that moving along…

Rob

5

So just for those following this thread, I contacted the help team for massive and they installed mrtrix/3.0.2 and also made certain cuda9.1 works with that module.

I have given it a first test, and it worked like a charm. It’s ridiculously fast. I have BIG files where the non-cuda-eddy needs like 5 hours, no kidding.
I completely missed the time it took using the eddy_cuda9.1, I could not have been longer than 3 minutes?

So for everybody who cares, if you like to do eddy_cuda on m3:
module load mrtrix/3.0.2

From my brief check it does not work for the other mrtrix modules.
Cheers
Mark