Dwiprepro with eddy cuda fails

#1

Dear experts
Maybe this is a question for fsl’s forum but any kind of help is welcomed.

I am trying to use dwipreproc with eddy_cuda on Ubuntu 16.04, but it fails.
I have installed the CUDA toolokit 8.0, renamed the eddy_cuda7.5 to eddy_cuda, copied in the fsl/5.0/bin and performed chmod 775 on it.
Note CUDA toolkit worked with their sample data and dwipreproc without -cuda option runs fine in 1h40 mins.
The system has VGA compatible controller: NVIDIA Corporation GM204GL [Quadro M4000] (rev a1) and uses “NVIDIA binary driver-version 375.66 from nvidia-375 (proprietary,tested)”.

u0061701@gbw-d-l0019:~/CI_study/OB01/prepro1$ dwipreproc dwi_deno_unring.mif dwi_deno_unring_epi_eddy.mif -rpe_header -se_epi b0_PA_AP.mif -cuda
dwipreproc: 
dwipreproc: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-help option) for more information.
dwipreproc: 
dwipreproc: Generated temporary directory: /home/luna.kuleuven.be/u0061701/CI_study/OB01/prepro1/dwipreproc-tmp-520HHY/
Command:  mrconvert /home/luna.kuleuven.be/u0061701/CI_study/OB01/prepro1/dwi_deno_unring.mif /home/luna.kuleuven.be/u0061701/CI_study/OB01/prepro1/dwipreproc-tmp-520HHY/dwi.mif
Command:  mrconvert /home/luna.kuleuven.be/u0061701/CI_study/OB01/prepro1/b0_PA_AP.mif /home/luna.kuleuven.be/u0061701/CI_study/OB01/prepro1/dwipreproc-tmp-520HHY/topup_in.mif
dwipreproc: Changing to temporary directory (/home/luna.kuleuven.be/u0061701/CI_study/OB01/prepro1/dwipreproc-tmp-520HHY/)
Command:  mrinfo dwi.mif -export_grad_mrtrix grad.b
Command:  mrconvert topup_in.mif topup_in.nii -stride -1,+2,+3,+4 -export_pe_table topup_datain.txt
Command:  topup --imain=topup_in.nii --datain=topup_datain.txt --out=field --fout=field_map.nii.gz --config=/usr/share/fsl/5.0/etc/flirtsch/b02b0.cnf
Command:  mrinfo dwi.mif -export_pe_eddy applytopup_config.txt applytopup_indices.txt
Command:  dwiextract dwi.mif -pe 0,1,0,0.0182 dwi_pe_1.nii
Command:  applytopup --imain=dwi_pe_1.nii --datain=applytopup_config.txt --inindex=1 --topup=field --out=dwi_post_topup.nii.gz --method=jac
Command:  mrconvert dwi_post_topup.nii.gz -grad grad.b - | dwi2mask - - | maskfilter - dilate - | mrconvert - mask.nii -datatype float32 -stride -1,+2,+3
Command:  mrconvert dwi.mif dwi.nii -stride -1,+2,+3,+4 -export_grad_fsl bvecs bvals -export_pe_eddy eddy_config.txt eddy_indices.txt
Command:  eddy_cuda --imain=dwi.nii --mask=mask.nii --acqp=eddy_config.txt --index=eddy_indices.txt --bvecs=bvecs --bvals=bvals --topup=field --out=dwi_post_eddy
dwipreproc: 
dwipreproc: [ERROR] Command failed: eddy_cuda --imain=dwi.nii --mask=mask.nii --acqp=eddy_config.txt --index=eddy_indices.txt --bvecs=bvecs --bvals=bvals --topup=field --out=dwi_post_eddy (dwipreproc:470)
dwipreproc: Output of failed command:
Entering EddyGpuUtils::LoadPredictionMaker

...................Allocated GPU # 0...................
thrust::system_error thrown in CudaVolume::common_assignment_from_newimage_vol after resize() with message: function_attributes(): after cudaFuncGetAttributes: invalid device function
terminate called after throwing an instance of 'thrust::system::system_error'
  what():  function_attributes(): after cudaFuncGetAttributes: invalid device function
dwipreproc: Changing back to original directory (/home/luna.kuleuven.be/u0061701/CI_study/OB01/prepro1)
dwipreproc: Script failed while executing the command: eddy_cuda --imain=dwi.nii --mask=mask.nii --acqp=eddy_config.txt --index=eddy_indices.txt --bvecs=bvecs --bvals=bvals --topup=field --out=dwi_post_eddy
dwipreproc: For debugging, inspect contents of temporary directory: /home/luna.kuleuven.be/u0061701/CI_study/OB01/prepro1/dwipreproc-tmp-520HHY/

Cheers,
Hamed

#2

Hey Hamed,

I have installed the CUDA toolokit 8.0, renamed the eddy_cuda7.5 to eddy_cuda, …

The fact that FMRIB provide a number of different versions of eddy for different versions of CUDA would suggest that compatibility is highly sensitive to version mismatches between compilation and runtime. So running a binary compiled against CUDA 7.5 when you have CUDA 8.0 installed is likely asking for trouble. I’d try downgrading your CUDA installation to 7.5 if possible.

Rob

1 Like
#3

Hi Rob,
I am still struggling with this issue.

I have 2 basic questions:

  1. Is it possible to use those binaries (from centos patch) on Ubuntu 16.04?
  2. Is my procedure i.e., “renaming of eddy_cuda7.5 to eddy_cuda, copying it in the fsl/5.0/bin and performing chmod 775 on it” correct?

Cheers,
Hamed

#4

I’ve never been able to get the CUDA binaries working myself; seemed to be because my video card is too old (GTS200 series), don’t know if you have the same issue.

  1. Would have thought so; but again, this is an FSL binary out of my control.
  2. The failure is coming from within eddy_cuda itself; if there were any issue with dwipreproc finding or executing the command, it would be reported as such. When sorting out issues such as this, you really are better off trying to run the raw command itself, rather than using some intermediate script, which then introduces uncertainty regarding where the issue is coming from.
1 Like