Invalid dwi scheme after dwifslpreproc

Hi MRtrix experts,

I have a problem using dwifslpreproc from the latest MRtrix version installed via conda on Linux Ubuntu 18.04.

I have dwi data with AP and PA phase encoding of all 64 diffusion directions and a single diffusion gradient strength (bval = 1000).
The files have 65 volumes (1 b0 volume + 64 directions)
The data is, converted from nifti to mif (mrconvert, adding the diffusion scheme bvec and bval files), denoised (dwidenoise) and gibbs ringing is removed (mrdegibbs).
Than the 2 dwi files (AP and PA) are concatenated,

mrcat degibbs_AP.mif degibbs_PA.mif degibbs_all.mif -axis 3

and submitted to dwifslpreproc.

dwifslpreproc degibbs_all.mif preproc.mif -rpe_all -pe_dir AP
-readout_time $(TotalReadoutTime)

The subsequent step in my pipeline is dwibiascorrect, but that fails with the error message,

[ERROR] Input image does not contain valid DW gradient scheme

Using mrinfo to look at the metadata of “preproc.mif” I get that it has 65 volumes as expected, but 130 entries in the dwi scheme.

With dwipreproc from an older MRtrix version (0.3.16) I don’t get this error.

Am I using dwifslpreproc wrong or is there a bug ?

best
Paul

There’s a chance this might relate to these changes, which were merged relatively recently, just before release.

Any chance you could post the full log from your dwifslpreproc run? You’d need to run it again with the -nocleanup option…

There’s a chance this might relate to these changes

Nope; my bad :raised_hand_with_fingers_splayed: :man_facepalming:

Due to me trying to be increasingly clever with the handling of image header key-value information in scripts, dwifslpreproc is currently erroneously discarding changes to the diffusion gradient table that occur during script execution. This is obviously most noticeable when using -rpe_all (or -rpe_header when explicit volume recombination occurs), as you end up with a gradient table with twice as many rows as it should have; but it also means that the motion-corrected bvecs table provided by eddy is currently not propagating to the output file.

Thanks for the report; I’ll push the fix to this and clean up a few other speed humps and hopefully we’ll push out 3.0.1 soon. In the meantime if you need to be able to proceed with your processing, this is the only change that you need to make to your local dwifslpreproc script file.

Rob

Hi there

I edited the dwifslpreproc script file on my Macbook pro as per your indication:

At least on the Macbook Pro it is producing errors. It finishes the eddyqc report but doest not output the final geomcorr.mif file, Just to be sure, I then changed the dwifslpreproc script back (i.e commenting out your edits) and it finished without error. I’ve copied the terminal output of both versions below. geomcorr_eddyqc3.mif should have been the output of your corrected dwifslpreproc script. geomcorr_eddyqc4.mif is the output of the script version that came with the MRtrix3 3.0.0 download:

The following command (that fails with the edited script) had finished previously without error:

 % dwifslpreproc degibbs.mif geomcorr_eddyqc3.mif -pe_dir pa -readout_time 0.0581166 -rpe_none -eddy_options " --repol" -eddyqc_text directory.eddy_qc3 -nocleanup

RERUN after R Smith's fix of code May 31, 2020
%  dwifslpreproc degibbs.mif geomcorr_eddyqc3.mif -pe_dir pa -readout_time 0.0581166 -rpe_none -eddy_options " --repol" -eddyqc_text directory.eddy_qc3 -nocleanup -force
dwifslpreproc: 
dwifslpreproc: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-help option) for more information.
dwifslpreproc: 
dwifslpreproc: [WARNING] Output file 'geomcorr_eddyqc3.mif' already exists; will be overwritten at script completion
dwifslpreproc: [WARNING] Output eddy QC directory already contains relevant files; these will be overwritten on completion
dwifslpreproc: Generated scratch directory: /Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/dwifslpreproc-tmp-MAS2V8/
Command:  mrconvert /Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/degibbs.mif /Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/dwifslpreproc-tmp-MAS2V8/dwi.mif -json_export /Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/dwifslpreproc-tmp-MAS2V8/dwi.json
dwifslpreproc: Changing to scratch directory (/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/dwifslpreproc-tmp-MAS2V8/)
Command:  dirstat dwi.mif -output asym
Command:  mrinfo dwi.mif -export_grad_mrtrix grad.b
Command:  dwi2mask dwi.mif - | maskfilter - dilate - | mrconvert - eddy_mask.nii -datatype float32 -strides -1,+2,+3
Command:  mrconvert dwi.mif -import_pe_table dwi_manual_pe_scheme.txt eddy_in.nii -strides -1,+2,+3,+4 -export_grad_fsl bvecs bvals -export_pe_eddy eddy_config.txt eddy_indices.txt
Command:  eddy_cuda7.5 --imain=eddy_in.nii --mask=eddy_mask.nii --acqp=eddy_config.txt --index=eddy_indices.txt --bvecs=bvecs --bvals=bvals --repol --out=dwi_post_eddy --verbose
dwifslpreproc: CUDA version of 'eddy' was not successful; attempting OpenMP version
Command:  eddy --imain=eddy_in.nii --mask=eddy_mask.nii --acqp=eddy_config.txt --index=eddy_indices.txt --bvecs=bvecs --bvals=bvals --repol --out=dwi_post_eddy --verbose
Command:  eddy_quad dwi_post_eddy -idx eddy_indices.txt -par eddy_config.txt -m eddy_mask.nii -b bvals
Command:  mrconvert dwi_post_eddy.nii.gz result.mif -strides -1,-2,3,4 -fslgrad dwi_post_eddy.eddy_rotated_bvecs bvals
Function: shutil.copy('dwi_post_eddy.eddy_parameters', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc3/eddy_parameters')
Function: shutil.copy('dwi_post_eddy.eddy_movement_rms', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc3/eddy_movement_rms')
Function: shutil.copy('dwi_post_eddy.eddy_restricted_movement_rms', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc3/eddy_restricted_movement_rms')
Function: shutil.copy('dwi_post_eddy.eddy_post_eddy_shell_alignment_parameters', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc3/eddy_post_eddy_shell_alignment_parameters')
Function: shutil.copy('dwi_post_eddy.eddy_post_eddy_shell_PE_translation_parameters', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc3/eddy_post_eddy_shell_PE_translation_parameters')
Function: shutil.copy('dwi_post_eddy.eddy_outlier_report', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc3/eddy_outlier_report')
Function: shutil.copy('dwi_post_eddy.eddy_outlier_map', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc3/eddy_outlier_map')
Function: shutil.copy('dwi_post_eddy.eddy_outlier_n_stdev_map', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc3/eddy_outlier_n_stdev_map')
Function: shutil.copy('dwi_post_eddy.eddy_outlier_n_sqr_stdev_map', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc3/eddy_outlier_n_sqr_stdev_map')

dwifslpreproc: [ERROR] Unhandled Python exception:
dwifslpreproc: [ERROR]   TypeError: Invalid input to run.function()
dwifslpreproc: [ERROR] Traceback:
dwifslpreproc: [ERROR]   /usr/local/bin/dwifslpreproc:1172 (in execute())
dwifslpreproc: [ERROR]     run.function(shutil.rmtree(os.path.join(eddyqc_path, 'quad')))
dwifslpreproc: [ERROR]   /usr/local/mrtrix3/lib/mrtrix3/run.py:472 (in function())
dwifslpreproc: [ERROR]     raise TypeError('Invalid input to run.function()')
dwifslpreproc: Scratch directory retained; location: /Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/dwifslpreproc-tmp-MAS2V8/
(base) jameshartzell@Jamess-MacBook-Pro 2nd.test.course.steps % 

RERUN AFTER CHANGING THE dwifslpreproc SCRIPT BACK TO THE WAY IT WAS:
(base) jameshartzell@Jamess-MacBook-Pro 2nd.test.course.steps % dwifslpreproc degibbs.mif geomcorr_eddyqc4.mif -pe_dir pa -readout_time 0.0581166 -rpe_none -eddy_options " --repol" -eddyqc_text directory.eddy_qc4 -nocleanup -force
dwifslpreproc: 
dwifslpreproc: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-help option) for more information.
dwifslpreproc: 
dwifslpreproc: [WARNING] Output file 'geomcorr_eddyqc4.mif' already exists; will be overwritten at script completion
dwifslpreproc: [WARNING] Output eddy QC directory already contains relevant files; these will be overwritten on completion
dwifslpreproc: Generated scratch directory: /Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/dwifslpreproc-tmp-T8RHSS/
Command:  mrconvert /Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/degibbs.mif /Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/dwifslpreproc-tmp-T8RHSS/dwi.mif -json_export /Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/dwifslpreproc-tmp-T8RHSS/dwi.json
dwifslpreproc: Changing to scratch directory (/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/dwifslpreproc-tmp-T8RHSS/)
Command:  dirstat dwi.mif -output asym
Command:  mrinfo dwi.mif -export_grad_mrtrix grad.b
Command:  dwi2mask dwi.mif - | maskfilter - dilate - | mrconvert - eddy_mask.nii -datatype float32 -strides -1,+2,+3
Command:  mrconvert dwi.mif -import_pe_table dwi_manual_pe_scheme.txt eddy_in.nii -strides -1,+2,+3,+4 -export_grad_fsl bvecs bvals -export_pe_eddy eddy_config.txt eddy_indices.txt
Command:  eddy_cuda7.5 --imain=eddy_in.nii --mask=eddy_mask.nii --acqp=eddy_config.txt --index=eddy_indices.txt --bvecs=bvecs --bvals=bvals --repol --out=dwi_post_eddy --verbose
dwifslpreproc: CUDA version of 'eddy' was not successful; attempting OpenMP version
Command:  eddy --imain=eddy_in.nii --mask=eddy_mask.nii --acqp=eddy_config.txt --index=eddy_indices.txt --bvecs=bvecs --bvals=bvals --repol --out=dwi_post_eddy --verbose
[Command:  eddy_quad dwi_post_eddy -idx eddy_indices.txt -par eddy_config.txt -m eddy_mask.nii -b bvals
Command:  mrconvert dwi_post_eddy.nii.gz result.mif -strides -1,-2,3,4 -fslgrad dwi_post_eddy.eddy_rotated_bvecs bvals
Function: shutil.copy('dwi_post_eddy.eddy_parameters', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc4/eddy_parameters')
Function: shutil.copy('dwi_post_eddy.eddy_movement_rms', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc4/eddy_movement_rms')
Function: shutil.copy('dwi_post_eddy.eddy_restricted_movement_rms', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc4/eddy_restricted_movement_rms')
Function: shutil.copy('dwi_post_eddy.eddy_post_eddy_shell_alignment_parameters', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc4/eddy_post_eddy_shell_alignment_parameters')
Function: shutil.copy('dwi_post_eddy.eddy_post_eddy_shell_PE_translation_parameters', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc4/eddy_post_eddy_shell_PE_translation_parameters')
Function: shutil.copy('dwi_post_eddy.eddy_outlier_report', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc4/eddy_outlier_report')
Function: shutil.copy('dwi_post_eddy.eddy_outlier_map', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc4/eddy_outlier_map')
Function: shutil.copy('dwi_post_eddy.eddy_outlier_n_stdev_map', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc4/eddy_outlier_n_stdev_map')
Function: shutil.copy('dwi_post_eddy.eddy_outlier_n_sqr_stdev_map', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc4/eddy_outlier_n_sqr_stdev_map')
Function: shutil.copytree('dwi_post_eddy.qc', '/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/directory.eddy_qc4/quad')
Command:  mrconvert result.mif /Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/geomcorr_eddyqc4.mif
dwifslpreproc: Changing back to original directory (/Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps)
dwifslpreproc: Scratch directory retained; location: /Users/jameshartzell/Documents/MRtrix3/S6_test/2nd.test.course.steps/dwifslpreproc-tmp-T8RHSS/

I’d be grateful for any guidance on this.

Cheers
James

Thanks.
That fixed it for me.

Hi Rob et al

Any further update/fix for this issue?
As written previously, the fix [https://github.com/MRtrix3/mrtrix3/pull/2067/files#diff-f5cadead80d3ce587cd3d0a4d149bcbeR1194) did not work on the Macbook Pro, so this has paused our analysis, pending a solution.

Cheers
James

Hi James,

Sorry for the delayed feedback.

That’s a separate issue; I don’t think the triggering of that problem is related to whether or not you manually fix the prior error, but rather whether or not the script is attempting to over-write an existing output directory. Fix is coming; code is here.

Cheers
Rob

Hi Rob

I ran the revised code (i.e. the revised dwifslpeproc with the first fix at 1192 and the second fix at 1172 ) and it finished without error and produced all the correct files.

Thanks very much.
Cheers
James