Dwipreproc eddy_cuda --slspec Unable to open file MultiBandGroups

1

Dear MRtrix Experts,

I have recently got a CUDA install up an running on our Linux Ubuntu 16.04 system. When running the following code:

dwipreproc {i}_in.mif.gz {i}_out.mif.gz -rpe_none -pe_dir PA -readout_time 0.0441 -eddy -cuda " --repol --mporder=18 --s2v_niter=10 --s2v_interp=trilinear --s2v_lambda=5 --slspec=json_slspec.txt "

I get the following error

dwipreproc: [ERROR] Command failed: eddy_cuda --imain=dwi.nii --mask=mask.nii --acqp=eddy_config.txt --index=eddy_indices.txt --bvecs=bvecs --bvals=bvals --repol --mporder=18 --s2v_niter=10 --s2v_interp=trilinear --s2v_lambda=5 --slspec=json_slspec.txt --out=dwi_post_eddy (dwipreproc:518)
dwipreproc: Output of failed command:
eddy: msg=MultiBandGroups::MultiBandGroups: Unable to open file
terminate called after throwing an instance of ‘EDDY::EddyException’
what(): eddy: msg=MultiBandGroups::MultiBandGroups: Unable to open file

when I remove the --slspec=<.txt> flag, eddy_cuda runs without this error.

Our data is from a Siemens 3T Trio, 36 volumes, Single Band, 72 interleaved slices. (Older sequence to preserve longitudinal acquisitions)

I suspect providing the interleaved slice acquisition order is important for slice to volume motion correction. Therefore, I am uneasy about moving forward without providing this information. Apologies, if this is an FSL specific issue, which I strongly suspect…

Any help would be very appreciated! Thank you all for your work!

Best, Derek

2

Hi Derek,

Most likely, the script is failing because you have provided a path to the “slspec” file that is relative to your current working directory, but then when eddy is actually invoked within dwipreproc the working directory is the temporary directory created by the script. Providing an absolute path to the slspec file may work.

I honestly don’t know what eddy will do if the --mporder option is specified but no slspec file is provided; personally I’d have expected it to not proceed.

The coming MRtrix3 update includes changes to dwipreproc and associated documentation for dealing with slice-to-volume correction. Once this change is pushed, the dwipreproc script will detect your use of “-eddy_options "--slspec="”, and copy that text file into the temporary directory where eddy will see it. Or more conveniently, MRtrix3 will capture slice timing information from DICOM and store it in the image header, and automatically generate the slspec file for eddy without requiring user intervention. :+1:

Rob

3

Hi Rob,

This seems like a very sensible explanation! Thank you. I was also surprised that eddy ran with --mporder but no slspec.txt file.

I have tried running dwipreproc and specifying the full directory to the slspec.txt however I either get the following error if I include a “/” at the start of the path

Command: eddy_cuda --imain=dwi.nii --mask=mask.nii --acqp=eddy_config.txt --index=eddy_indices.txt --bvecs=bvecs --bvals=bvals --repol --mporder=18 --s2v_niter=10 --s2v_interp=trilinear --s2v_lambda=5 –slspec=/data/users/DSA/EDDYTEST/my_slspec.txt --out=dwi_post_eddy
Traceback (most recent call last):
File “/data/bin/mrtrix3/bin/dwipreproc”, line 518, in
run.command(eddy_cmd + ’ --imain=dwi.nii --mask=mask.nii --acqp=eddy_config.txt --index=eddy_indices.txt --bvecs=bvecs --bvals=bvals’ + eddy_in_topup_option + eddy_manual_options + ’ --out=dwi_post_eddy’)
File “/data/bin/mrtrix3/lib/mrtrix3/run.py”, line 175, in command
stdout_text = tempfiles[index][0].read().decode(‘utf-8’)
File “/usr/lib/python2.7/encodings/utf_8.py”, line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0xa0 in position 0: invalid start byte

or the same ‘msg=MultiBandGroups: Unable to open file error’ as before if not including the “/” i.e. –slspec=data/users/DSA/EDDYTEST/my_slspec.txt

Thus, it does not look like I will be able to provide the full path to my slspec.txt file within dwipreproc.

I’m looking forward to the updates you detailed! How imminent are these going to being available? In the meantime do you see a workaround? Is it possible to copy slspec.txt into the dwipreproc temporary directory and ‘restart’ dwipreproc from there?

Thank you again!

Derek

4

UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0xa0 in position 0: invalid start byte

Okay, that looks like a similar issue to Hamed’s recent post and the corresponding GitHub issue. The MRtrix3 Python library is falling over when trying to read the text written to the terminal by eddy; what I don’t know is whether or not the data it is failing to decode is healthy feedback or an error message. I’ll see if I can reproduce it.

How imminent are these going to being available?

I’ve been pushing hard on this; was aiming for the end of the week but there’s likely to be ongoing discussion about this particular issue. You can see the outstanding issues via the relevant milestone and pull request on GitHub.

In the meantime do you see a workaround? Is it possible to copy slspec.txt into the dwipreproc temporary directory and ‘restart’ dwipreproc from there?

That should be possible. All of the scripts that use the MRtrix3 Python scripts have the -continue option; it’s quite clunky and doesn’t work in all scenarios, but in this case it should be able to serve that purpose.

Alternatively, if you’re not averse to some version control shenanigans, the development branch that includes the updates to dwipreproc described earlier is publicly accessible, so you’re welcome to give it a try (bearing in mind that checking out that branch will come with a large batch of other changes throughout the entire software).

Rob

5

Hi Rob

This would be awesome. But do you think this would be an option for Philips DICOMs as well? I guess they don’t include this information. Do they?

Cheers,
Hamed

6

Hi Derek,

I have similar Linux Ubuntu 16.04 system.

$ uname -m && cat /etc/*release
x86_64
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION=“Ubuntu 16.04.4 LTS”
NAME=“Ubuntu”
VERSION=“16.04.4 LTS (Xenial Xerus)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 16.04.4 LTS”
VERSION_ID=“16.04”
HOME_URL=“http://www.ubuntu.com/”
SUPPORT_URL=“http://help.ubuntu.com/”
BUG_REPORT_URL=“Bugs : Ubuntu”
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

with nvidia:

$ nvidia-smi
Thu Apr 12 15:13:02 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 390.30 Driver Version: 390.30 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro M4000 Off | 00000000:03:00.0 On | N/A |
| 46% 36C P8 13W / 120W | 416MiB / 8118MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1255 G /usr/lib/xorg/Xorg 297MiB |
| 0 2682 G compiz 109MiB |
| 0 8425 G unity-control-center 3MiB |
±----------------------------------------------------------------------------+

and

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17

But so far unable to use eddy cuda.
Could you please explain more about how you made eddy cuda work.
Thanks and cheers,
Hamed

7

Hi Derek

Never mind my question.

Apparently, eddy_cuda binary is included in fsl 5.0.11 and I didn’t know that I had to call -cuda explicitly in the dwipreproc command (I thought it will find it automatically!). :face_with_raised_eyebrow:

So I did that and my 102d multishell topup+eddy_openmp (8+35 minutes) runtime reduced to (8+15 minutes)! Nice! :slight_smile:

Cheers,
Hamed

8

Hi Rob,

Your sensible explanations are once again helping keep me sane!

I have tried using the -continue option as you suggested. This appears to get around the msg=MultiBandGroups: Unable to open file error but still returns UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0x9e in position 1: invalid start byte

Therefore, I think adding the full path to the slspec.txt file for the --slspec flag might have worked, however the file “my_slspec.txt” or “myslspec.txt” is failing to be read regardless.

From reviewing Hamed’s recent post and the corresponding GitHub issue a solution did not jump out at me. Did I miss something?

It seems like the update for this should hopefully be available soon (days not weeks) :crossed_fingers::smile: so I’m thinking my best option may be to wait it out. Although, this may not solve the unicode issue…

9

But do you think this would be an option for Philips DICOMs as well? I guess they don’t include this information. Do they?

Correct. From the data I have at hand, Philips never include this information. Conversely, GE provide the acquisition time of each slice as being identical to the time of the commencement of the acquisition. In both cases MRtrix3 will detect that there is no timing contrast between slices, and hence not write any slice timing information to the image header. Perhaps I need to be more conservative by reporting that this capability applies for Siemens data only…

I’d also encourage Philips and GE users to put pressure on their vendors to push software updates to enable this. I’m surprised the fMRI community haven’t already done that, given the availability of software for slice timing correction.

… I didn’t know that I had to call -cuda explicitly in the dwipreproc command (I thought it will find it automatically!)

Yes, currently it explicitly requires setting the -cuda option. At the time that script was first implemented I expected very few users to have gone to the effort of getting CUDA working, but many may have downloaded both binaries, and hence attempting to execute the CUDA version would lead to failures.

In the 3.0_RC3 update, detection of the CUDA eddy binary is more robust (searches for the highest CUDA version available, so you don’t have to explicitly construct an “eddy_cuda” softlink). I also now have mechanisms for catching failed command executions and allowing the script to continue. So I wouldn’t be against modifying dwipreproc to use the CUDA version by default if it can be found, if that would lead to less user issues / confusion. Thoughts?

This … still returns UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0x9e in position 1: invalid start byte

I’m trying to duplicate your use case as a way for me to simultaneously investigate the UTF-8 encoding issue. The only insight I’ve had so far has been in certain circumstances where the CUDA version of eddy generates a segmentation fault: the command outputs text relating to the memory areas implicated, but somewhere in that text is raw binary data that results in invalid codes if interpreted as UTF-8. Unfortunately this is not reproducible, which is always the first step in diagnosing and fixing faults.

… I’m thinking my best option may be to wait it out. Although, this may not solve the unicode issue…

Given this has cropped up a number of times now, the MRtrix3 update won’t be pushed out without some solution to the Unicode issue. Worst case scenario, terminal text output by some commands may not appear precisely as intended; but that’s a helluva lot better than users getting unhandled Python exception messages. Just trying to generate good reproducible use cases so that I can figure out the best solution. And make sure the solution works on all 3 OS’s, both versions of Python on each, and with and without the -debug option for each :exploding_head:

10

Hi Derek

I explain what happens to a couple of my subjects. It may or may not solve your issue but could be helpful for others with the same error.
In the dwipreproc temporary directory, there is a log file. In one of the lines in the middle, there is this command (in my case):

dwiextract dwi.mif -pe 0,1,0,0.0173 - | mrconvert - dwi_pe_1.nii -json_export dwi_pe_1.json

which creates a .json file. If I open this file (for some of my subjects), I see that there is “È” (E with a grave accent) or something like that in the subject’s name (coming originally from DICOMs/scanner) which in later steps causes ‘utf8’ problem when the .json file is used by:

mrconvert dwi_pe_1_topup.nii.gz dwi_pe_1_topup.mif -json_import dwi_pe_1.json

Traceback (most recent call last):
File “/home/…/mrtrix3/bin/dwipreproc”, line 488, in
run.command('mrconvert ’ + temp_path + ’ ’ + output_path + ’ -json_import ’ + json_path)
File “/home/…/mrtrix3/lib/mrtrix3/run.py”, line 183, in command
stderr_text = tempfiles[index][1].read().decode(‘utf-8’)
File “/usr/lib/python2.7/encodings/utf_8.py”, line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0xc9 in position 228: invalid continuation byte
dwipreproc:

So what I do is to replace this letter with normal ‘E’ (without accent grave) in the .json file, save the file and continue the dwipreproc as where it crashed.

Cheers,
Hamed

11

Check this out please:
http://dbic.dartmouth.edu/wiki/index.php/Slice_Acquisition_Order
Using above and knowing the ‘acquisition’ type from the DICOMs’ header (luckily they have included this :slight_smile: ), I guess it would be possible to address this issue.?

Cheers,
Hamed

12

I explain what happens to a couple of my subjects. It may or may not solve your issue but could be helpful for others with the same error.

While the reported error is the same, the underlying issue that is leading to the error differs quite significantly between your two cases (more info on GitHub).

@DAndrews: The way I was able to “reproduce” your issue was by providing an invalid slspec file, which leads to a CUDA segmentation fault; that segmentation fault then (sometimes) contains non-text data that causes the script library to have a hiccup. So in the absence of you actually seeing that error, I would suggest checking over your slpec file extremely closely; eddy does not appear to currently perform a sanity check on it, so if e.g. a slice is missing from the list, that will lead to your currently-rather-useless error feedback.

Check this out please:

I’ll have a look into that. I’m aware of the slice acquisition order fields in NIfTI, but I’d need to figure out how dcmstack is extracting that information from DICOM.

13

Ok, I went back and looked at my slspec.txt file. I had made a poor assumption that our slice acquisition started with slice “1”, when it actually starts with slice “0”.

After adjusting the slspec file, and taking the steps to copy the slspec file into the temp folder and use -continue dwipreproc and eddy_cuda run with no errors.

My test results look great! Thank you again Rob for all your help!

Best, Derek

14

Hi Rob,

Unfortunately, my celebration may have been a bit premature. While eddy_cuda appears to run and move on to the following mrconvert steps, then I get a general eddy_cuda failed error…

Command: eddy_cuda --imain=dwi.nii --mask=mask.nii --acqp=eddy_config.txt --index=eddy_indices.txt --bvecs=bvecs --bvals=bvals --repol --mporder=18 --s2v_niter=10 --s2v_interp=trilinear --s2v_lambda=5 --slspec=json_slspec.txt --out=dwi_post_eddy

Command: mrconvert dwi_post_eddy.nii.gz result.mif -stride -1,2,3,4 -fslgrad dwi_post_eddy.eddy_rotated_bvecs bvals

Command: mrconvert result.mif /data/proj/app/pipeline/DWI_TH/100604-200_3/100604-200_3_posteddy.mif.gz

dwipreproc: Changing back to original directory (/data/proj/app/pipeline/DWI_TH/100604-200_3)

dwipreproc: Script failed while executing the command: eddy_cuda --imain=dwi.nii --mask=mask.nii --acqp=eddy_config.txt --index=eddy_indices.txt --bvecs=bvecs --bvals=bvals --repol --mporder=18 --s2v_niter=10 --s2v_interp=trilinear --s2v_lambda=5 --slspec=json_slspec.txt --out=dwi_post_eddy

dwipreproc: For debugging, inspect contents of temporary directory: /data/proj/app/pipeline/DWI_TH/100604-200_3/dwipreproc-tmp-7ZN2XE

the output file is produced and looks good.

The error.txt file in the tmp directory gives the same old ‘msg=MultiBandGroups: Unable to open file error’ which I’m assuming is a hold over from the first run of dwipreproc that creates the tmp directory, which I then use -continue to flag. I’m also thinking this error.txt could be the reason for this error warning, when in fact all is well? I am using the -nocleanup flag…

15

The error.txt file in the tmp directory gives the same old msg=MultiBandGroups: Unable to open file error which I’m assuming is a hold over from the first run of dwipreproc that creates the tmp directory, which I then use -continue to flag. I’m also thinking this error.txt could be the reason for this error warning, when in fact all is well? I am using the -nocleanup flag…

Correct; all is indeed well. I only recently discovered this issue myself. It’s already fixed for the coming update.

16

Correct; all is indeed well.

Great news! I figured that was the issue. Best wishes with the rollout of the new update. Your guidance is very very appreciated!

17

Hi,
joining the ranks of problems occuring in dwipreproc associated with eddy_cuda. Since I’ve updated the latest MRtrix release candidate AND my macOS to High Sierra 10.13.4 (in one go) a couple of days ago, an error has kept occuring when running dwipreproc (the exact command was dwipreproc dwi_denoised.mif dwi_den_preproc.mif -rpe_pair -se_epi b0s.mif -pe_dir AP -eddy_options " --data_is_shelled --slm=linear ; and was run on unanonymized data), telling me that eddy_cuda_6.5 could not be carried out since I did not have the necessary permissions. I then reinstalled my FSL. Now I get the following message running the same command:

[WARNING] Command failed: eddy_cuda --imain=eddy_in.nii --mask=eddy_mask.nii --acqp=eddy_config.txt --index=eddy_indices.txt --bvecs=bvecs --bvals=bvals --topup=field --data_is_shelled --slm=linear --out=dwi_post_eddy
dwipreproc: [WARNING] CUDA version of eddy appears to have failed; trying OpenMP version

I also went back to a dataset I had preprocessed before the updates (without any warnings), using the exact same command as above. Now I’m getting the same warning there. Does anyone have any idea what the issue might be? Do you think that’s a problem of the MRtrix update or the OSX?

Best,
Marlene.

18

This sounds like an incompatibility problem between eddy and your OS and cuda similar to here. Does eddy_cuda work on any data? You might need to update cuda.

19

I had the same eddy_cuda error running dwipreproc on Amazon web services a couple days ago. I got around it by reinstalling cuda 7.5 toolkit and older nvidia-367 drivers which supported the GPU on the aws instance. Worked “almost” like a charm after that…

I had the Unicode error on 3 of 10 subjects during dwipreproc. It was a problem when mrtrix was reading the header. I had an older version of mrtrix still running on host machine (without cuda). So I ran dwipreproc and terminated it after it started the non-cuda Eddy. I just rsnyc’d the temp directory that dwipreproc created to AWS and manually executed the remaining dwipreproc commands (eddy_cuda, some mrconvert commands, and copying data from tmp directory back to original data folder). Annoying, but worked. I’ll have to see if the -continue switch works for me. That seems like an easier work around than what I did.

20

@maxpietsch I updated cuda, but the problem remains. Unfortunately, eddy_cuda doesn’t seem to work on any of my data any more.
@sniogi Which operating system are you running? The cuda 7.5 toolkit is not available for OSX 20.13.4, which I am now using. Neither are any of the nvidia-367 drivers I found. Do you have other suggestions?