[ERROR] Unable to determine matching DWI volume pairs for reversed phase-encode combination

I have a diffusion set collected as part of the myconnectome (Long-term neural and physiological phenotyping of a single human | Nature Communications) acquisition.

The data are 2 full difffusion-sets, one acquired with a R->L phase-encoding, and the other in the opposite direction (L->R). However, dwipreproc with -rpe_all doesn’t recognize the matching gradients/volumes across the 2 sets:

I’m running the following code (see myconnectomeDWI/scripts/advpreproc at master · AlistairPerry/myconnectomeDWI · GitHub):

mrcat rawdataRLdn.mif rawdataLRdn.mif allDWIs.mif -axis 3 -force -nthreads $ncpus

dwipreproc -rpe_all -export_grad_mrtrix adjencoding.b -pe_dir RL allDWIs.mif eddycorr.mif -eddy_options " --repol --data_is_shelled --mb=3 --ol_type=both " -nthreads $ncpus -force -tempdir . -nocleanup -debug

Which produces the following error:

dwipreproc: [DEBUG] phaseEncoding.dir() (from dwipreproc:167): rl → [-1, 0, 0]
dwipreproc: [ERROR] Unable to determine matching DWI volume pairs for reversed phase-encode combination

This is very strange as I have embedded the same gradient encoding to the RL and LR sets, and manually set the PhaseEncodingDirection:

mrconvert rawdataLR.mif rawdataLRgrad.mif -grad rawencoding.b -set_property PhaseEncodingDirection i -nthreads $ncpus -force 
mrconvert rawdataRL.mif rawdataRLgrad.mif -grad rawencoding.b -set_property PhaseEncodingDirection i- -nthreads $ncpus -force

The data were unfortunately extracted as compressed nii's (.nii.gz), and hence the json files are not embedded.

Checking the image headers nevertheless suggests nothing wrong:

[alistaiP@hpcpbs01 preproc]$ mrinfo rawdataRLdn.mif

Dimensions: 128 x 128 x 78 x 64
Voxel size: 1.74219 x 1.74219 x 1.7 x 5
Data strides: [ 1 2 3 4 ]
Format: MRtrix
Data type: 32 bit float (little endian)
Intensity scaling: offset = 0, multiplier = 1
Transform: 1 0 0 -109.8
-0 1 0 -120
-0 0 1 -3.101
PhaseEncodingDirection: i-

[alistaiP@hpcpbs01 preproc]$ mrinfo rawdataLRdn.mif

Dimensions: 128 x 128 x 78 x 64
Voxel size: 1.74219 x 1.74219 x 1.7 x 5
Data strides: [ 1 2 3 4 ]
Format: MRtrix
Data type: 32 bit float (little endian)
Intensity scaling: offset = 0, multiplier = 1
Transform: 1 0 0 -109.8
-0 1 0 -120
-0 0 1 -3.101
PhaseEncodingDirection: i

The dwipreproc python script (lines 223-235) is looking for the number of matching gradient pairs to equal half the number of volumes in the dwi series:

elif PE_design == ‘All’:
grad_matchings = [ num_volumes ] * num_volumes
grad_pairs =
for index1 in range(num_volumes):
if grad_matchings[index1] == num_volumes: # As yet unpaired
for index2 in range(index1+1, num_volumes):
if grad_matchings[index2] == num_volumes: # Also as yet unpaired
if grad[index1] == grad[index2]:
grad_matchings[index1] = index2;
grad_matchings[index2] = index1;
grad_pairs.append([index1, index2])
if not len(grad_pairs) == num_volumes/2:
app.error(‘Unable to determine matching DWI volume pairs for reversed phase-encode combination’)

1 Like

Alistair

The volume recombination code in dwipreproc is pretty bugged at the moment; can you give it a try with the MRtrix3 code branch tag_3.0_RC2?

Rob

@rsmith can do,

Unfortunately, can’t do it just yet as I’m running a fair number of jobs.

Will switch the branch once they’re done.

Alistair

@rsmith,

Still appears to be no luck.

To confirm that I am using the branch/version which you intended:

[alistaiP@hpcnode005 preproc]$ mrinfo allDWIscheck.mif
mrtrix_version: 3.0_RC1-176-g73e60f22

[alistaiP@hpcnode005 preproc]$ dwipreproc
MRtrix 3.0_RC1-176-g73e60f22 dwipreproc bin version: 3.0_RC1-176-g73e60f22

Alistair

MRtrix 3.0_RC1-176-g73e60f22 dwipreproc bin version: 3.0_RC1-176-g73e60f22

No, that code is a month old; the relevant changes were merged into the tag_3.0_RC2 branch two weeks ago. Try a git pull to get the most recent changes on that branch onto your local machine.

Apologies, I am working on a cluster atm.

I’m now on 3.0_RC1-194-g317fd642, with the branch tag_3.0_RC2.

Still no luck.

Alistair

Can you confirm that it’s still exactly the same error? I.e.:

dwipreproc: [ERROR] Unable to determine matching DWI volume pairs for reversed phase-encode combination

You should at least see that the block of dwipreproc code you originally copy&pasted has changed. If it’s still the same error, and you’re absolutely certain that the two halves of the gradient table are identical, I probably need to get access to the file to try to reproduce.

Exactly the same error: For confirmation see:

[alistaiP@hpcpbs01 mrtrix3]$ git branch

  • tag_3.0_RC2

And the dwipreproc version:

[alistaiP@hpcpbs01 mrtrix3]$ ./dwipreproc
MRtrix 3.0_RC1-194-g317fd642 dwipreproc

Which has the following code:

elif PE_design == ‘All’:
grad_matchings = [ num_volumes ] * num_volumes
grad_pairs =
for index1 in range(num_volumes):
if grad_matchings[index1] == num_volumes: # As yet unpaired
for index2 in range(index1+1, num_volumes):
if grad_matchings[index2] == num_volumes: # Also as yet unpaired
if grad[index1] == grad[index2]:
grad_matchings[index1] = index2;
grad_matchings[index2] = index1;
grad_pairs.append([index1, index2])
if not len(grad_pairs) == num_volumes/2:
app.error(‘Unable to determine matching DWI volume pairs for reversed phase-encode combination’)

Ill get about cloudstoring you the data.

Alistair

The code should look like this:

  elif PE_design == 'All':
    if num_volumes%2:
      app.error('If using -rpe_all option, input image must contain an even number of volumes')
    grads_matched = [ num_volumes ] * num_volumes
    grad_pairs = [ ]
    for index1 in range(int(num_volumes/2)):
      if grads_matched[index1] == num_volumes: # As yet unpaired
        for index2 in range(int(num_volumes/2), num_volumes):
          if grads_matched[index2] == num_volumes: # Also as yet unpaired
            if grads_match(grad[index1], grad[index2]):
              grads_matched[index1] = index2;
              grads_matched[index2] = index1;
              grad_pairs.append([index1, index2])
              break
        else:
          app.error('Unable to determine matching reversed phase-encode direction volume for DWI volume ' + index1)
    if not len(grad_pairs) == num_volumes/2:
      app.error('Unable to determine complete matching DWI volume pairs for reversed phase-encode combination')

Are you sure you don’t have multiple MRtrix3 installations and are querying / executing the wrong one?

@rsmith,

Apologies, there was some other installation interfering. Weird.

Regardless, I’ve switched machines and am reproducing a similar error:

Traceback (most recent call last):
  File "/opt/mrtrix3-020817/bin/dwipreproc", line 256, in <module>
    app.error('Unable to determine matching reversed phase-encode direction volume for DWI volume ' + index1)
TypeError: cannot concatenate 'str' and 'int' objects

The newish error string reveals I’m on the correct version/branch.

Anyway, ill still send the data over.

Alistair

OK, thanks for the data.

Fairly simple problem: When I reworked the code for performing that explicit volume matching, I failed to take into account that sometimes b=0 volumes get encoded with a zero-length vector, rather than the conventional [0, 0, 1].

So while the fix is on its way, if you manually edit your gradient table to have b=0 volumes “pointing” in the z-direction, the existing code on the tag_3.0_RC2 branch should then work.

Rob

Hey @rsmith ,

Can you confirm the data were able to be fully processed through?

I’m still receiving the same error (with the same data):

[ERROR] Unable to determine matching DWI volume pairs for reversed phase-encode combination

And the version I’m quite sure is correct (unless again there is something else interfering):

[alistaiP@hpcpbs01 dwipreproc-tmp-GI9USH]$ mrinfo dwi.mif 
Image:               "dwi.mif"                    
mrtrix_version:    3.0_RC1-198-g166cc5ac

I processed your data, though only with including a fix for a separate issue (FSL commands ignoring the FSLOUTPUTTYPE variable), which you may or may not have (most likely not since nobody else has encountered / reported it.

166cc5ac is the current head on the master branch; you want to be on the tag_3.0_RC2 branch, which is currently 3aefda4. Though beware that you want to be checking the version number of the actual command you’re running: The mrtrix_version flag you’re looking at there is the version of MRtrix3 that was used to generate that image, and is stored in the image header; it’s not the version of MRtrix3 that you’re invoking with your mrinfo call.

I typically manually define the FSLOUTPUTTYPE, so no issues here.

Yes, all good now. Data appears to have now passed that erroneous stage.

I note that It looks that the tag_3.0_RC2 branch will soon be merged into master (Tag 3.0 RC2 by Lestropie · Pull Request #1052 · MRtrix3/mrtrix3 · GitHub).

Best,

Alistair

@rsmith, False alarm. Another issue has appeared.

For one of the subcommands in dwipreproc:

mrcat dwi_pe_1_topup.mif dwi_pe_2_topup.mif - | dwi2mask - - | maskfilter - dilate - | mrconvert - mask.nii -datatype float32 -stride -1,+2,+3

The following error appears:

mrconvert: [ERROR] malformed PE scheme in image “mask.nii” - number of rows does not equal number of volumes

Obviously there is something corrupt in the embedded .mif files which is being passed across the - pipes. No issues are encountered if the data is piped to local *.nii files. For example:

[alistaiP@hpcnode031 dwipreproc-tmp-PIOII1]$ mrcat dwi_pe_1_topup.mif dwi_pe_2_topup.mif - | dwi2mask - - | maskfilter - dilate testdilate.nii -force | mrconvert testdilate.nii mask.nii -datatype float32 -stride -1,+2,+3 -force

Here is the full debug from the initial mrconvert:

maskfilter: [100%] applying dilate filter to image -
mrconvert: [DEBUG] reading key/value file “/tmp/mrtrix-tmp-bP2Q4T.mif”…
mrconvert: [DEBUG] sanitising image information…
mrconvert: [DEBUG] mapping piped image “/tmp/mrtrix-tmp-bP2Q4T.mif”…
mrconvert: [DEBUG] memory-mapping file “/tmp/mrtrix-tmp-bP2Q4T.mif”…
mrconvert: [DEBUG] file “/tmp/mrtrix-tmp-bP2Q4T.mif” mapped at 0x7fa0413b1000, size 159744 (read-only)
mrconvert: [DEBUG] image “/tmp/mrtrix-tmp-bP2Q4T.mif” loaded
mrconvert: [DEBUG] image “/tmp/mrtrix-tmp-bP2Q4T.mif” initialised with strides = [ 1 128 16384 ], start = 0, using indirect IO
mrconvert: [INFO] creating image “mask.nii”…
mrconvert: [DEBUG] sanitising image information…
mrconvert: [DEBUG] creating empty file “mask.nii”
mrconvert: [DEBUG] resizing file “mask.nii” to 5112160
mrconvert: [DEBUG] sanitising image information…
mrconvert: [DEBUG] memory-mapping file “mask.nii”…
mrconvert: [DEBUG] “mask.nii” appears to reside on a networked filesystem - using delayed write-back
mrconvert: [DEBUG] file “mask.nii” held in RAM at 0x7fa03f1f0010, size 5111808
mrconvert: [DEBUG] image “mask.nii” loaded
mrconvert: [DEBUG] image “mask.nii” initialised with strides = [ 1 128 16384 ], start = 0, using indirect IO
mrconvert: [INFO] writing back contents of mapped file “mask.nii”…
mrconvert: [DEBUG] image “mask.nii” unloaded
mrconvert: [DEBUG] unmapping file “/tmp/mrtrix-tmp-bP2Q4T.mif”
mrconvert: [DEBUG] deleting piped image file “/tmp/mrtrix-tmp-bP2Q4T.mif”…
mrconvert: [DEBUG] image “/tmp/mrtrix-tmp-bP2Q4T.mif” unloaded
mrconvert: [ERROR] malformed PE scheme in image “mask.nii” - number of rows does not equal number of volumes

:pray: Let’s pray for it. I understand that our users are in favour of a fast(er) release cycle? Do voice your opinion if that’s the case! :smiley:

I note that It looks that the tag_3.0_RC2 branch will soon be merged into master

Those changes have been merged into master today. I’m currently re-running your data on up-to-date code, and it’s made it to the eddy stage, which would suggest it’s made it past the line where you’ve quoted the most recent error:

mrcat dwi_pe_1_topup.mif dwi_pe_2_topup.mif - | dwi2mask - - | maskfilter - dilate - | mrconvert - mask.nii -datatype float32 -stride -1,+2,+3
mrconvert: [ERROR] malformed PE scheme in image “mask.nii” - number of rows does not equal number of volumes

My suspicion with this one is that when you checked out the tag_3.0_RC2 branch in order to test the changes to dwipreproc, you did not additionally re-build the MRtrix3 binaries. Included in the changes associated with dwipreproc to get -rpe_all working again were also changes to how the phase-encoding table is handled; this includes dwi2mask, which now appropriately erases the phase-encoding table (since its rows no longer correspond to image volumes), preventing warnings / errors in subsequent steps. So I’m hoping that by re-building your binaries, your dwipreproc run should complete.

Having tremendous fun with dwipreproc as always :weary:
Rob

Oh yes I wasn’t doing this…

Yes, have now git pulled, and built the latest binaries on the master branch.

Have some solace in that the data has now passed through with no issues :slight_smile:

Thanks again,

Alistair