Dwi2response with tournier

malav · June 13, 2016, 10:25am

Respected sir/ma’am,

For performing tractography, I wish to execute dwi2mask, dwi2response, dwi2fod one after another. But unfortunately I am stuck at dwi2response. I am trying to execute it with tournier, according to the document on this link.

I do not get any error stating that the no. of arguements is less than necessity, but I get error regarding mrconvert function, which I am supposing would be performed eventually for the dwi2response.
Please find the code and the error shown below.

mallo@fountainhead:~/mrtrix3/scripts$ ./dwi2response tournier /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/nii2mif.mif /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/response.txt
dwi2response: 
dwi2response: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-help option) for more information.
dwi2response: 
dwi2response: Generated temporary directory: /tmp/dwi2response-tmp-TYVDLT/
Command: mrconvert /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/nii2mif.mif - -stride 0,0,0,1 | dwiextract - /tmp/dwi2response-tmp-TYVDLT/dwi.mif
dwiextract: [ERROR] no filename supplied to standard input (broken pipe?)
dwiextract: [ERROR] error opening image "-"
dwi2response: [ERROR] Command failed: mrconvert /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/nii2mif.mif - -stride 0,0,0,1 | dwiextract - /tmp/dwi2response-tmp-TYVDLT/dwi.mif
dwi2response: Changing back to original directory (/home/mallo/mrtrix3/scripts)
dwi2response: Deleting temporary directory /tmp/dwi2response-tmp-TYVDLT/
mallo@fountainhead:~/mrtrix3/scripts$

Please help me rectify this issue.

rsmith · June 14, 2016, 12:41am

Hi @malav,

I’m not quite sure what’s going on there: dwiextract is failing because it’s not receiving the temporary image output by mrconvert, but mrconvert itself is not providing any indication regarding how or why it has failed.

Are you able to simply run mrconvert on the file nii2mif.mif?

Also, can you please provide your Python and MRtrix3 version information. The mechanism within the Python script back-end responsible for execution of commands was modified on 30 May, so I need to know whether or not those changes are introducing an issue.

Thanks
Rob

jdtournier · June 14, 2016, 9:12am

It looks to me like mrconvert crashed out, and hence didn’t provide any input for dwiextract to process, which would explain the messages that we do see (the shell typically won’t report a segfault if the command is not last in the pipeline). The most likely reason for mrconvert to crash is an unhandled memory allocation failure, due to running out of RAM, as has been reported a few times recently (here, here, and here).

Just to check this: how much RAM does your system have, and how big are the images you’re trying to process…?

malav · June 14, 2016, 9:25am

HI @rsmith and @jdtournier,

Thanks for looking into the matter.

My device RAM is 8 GB, and I am trying to run these operations with a dwi.mif file having size of 2.2 GB.
The version of the tools are as mentioned below.

mallo@fountainhead:/usr/bin$ python -V
Python 2.7.6

mallo@fountainhead:~/mrtrix3$ mrview -v
mrview 0.2.11 (revision 373)
Author: J-Donald Tournier (d.tournier@brain.org.au)
Copyright 2008 Brain Research Institute, Melbourne, Australia
using MRtrix 0.2.11, glib 2.40.2, GSL 1.16 (build Oct 22 2013)

Fortunately, (I actually don’t know how this happened) now I am able to run the same command without the same error,
but unfortunately, I am facing other error.

mallo@fountainhead:~/mrtrix3/scripts$ ./dwi2response tournier /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/dwi.mif /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/response.txt
dwi2response:
dwi2response: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-help option) for more information.
dwi2response:
dwi2response: Generated temporary directory: /tmp/dwi2response-tmp-1TFTG4/
Command: mrconvert /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/dwi.mif - -stride 0,0,0,1 | dwiextract - /tmp/dwi2response-tmp-1TFTG4/dwi.mif
dwiextract: [ERROR] invalid diffusion gradient table dimensions
dwi2response: [ERROR] Command failed: mrconvert /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/dwi.mif - -stride 0,0,0,1 | dwiextract - /tmp/dwi2response-tmp-1TFTG4/dwi.mif
dwi2response: Changing back to original directory (/home/mallo/mrtrix3/scripts)
dwi2response: Deleting temporary directory /tmp/dwi2response-tmp-1TFTG4/

I am guessing that while running the dwi2response, it will be running the mrconvert command for specific format of file to access it in its native way, but during this conversion, it’s facing some problems with gradient dimensions of the dwi.mif file (either it’s not available in the header or not in the specific way it should be).

Please help me solve this.

jdtournier · June 14, 2016, 9:49am

OK, it looks like your mrview command is from the previous version of MRtrix (0.2.11). I was expecting to see something like:

$ mrview -version
== mrview 0.3.14-693-g282d2505 ==
64 bit release version, built Jun 10 2016, using Eigen 3.2.8
Author(s): J-Donald Tournier (d.tournier@brain.org.au), Dave Raffelt (david.raffelt@florey.edu.au) and Robert E. Smith (r.smith@brain.org.au)
Copyright (c) 2008-2016 the MRtrix3 contributors

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/

MRtrix is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

For more details, see www.mrtrix.org

What I’m worried about is that if the old MRtrix 0.2 commands are in your PATH, they will be used instead and the script will likely fail since things have changed quite a bit… So to verify this, check that your version of mrconvert is correct (the output below is what my system reports):

$ which mrconvert
/home/jdt13/mrtrix3/release/bin/mrconvert

and

$ mrconvert -version
== mrconvert 0.3.14-693-g282d2505 ==
64 bit release version, built Jun 10 2016, using Eigen 3.2.8
Author(s): J-Donald Tournier (jdtournier@gmail.com)
Copyright (c) 2008-2016 the MRtrix3 contributors

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/

MRtrix is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

For more details, see www.mrtrix.org/home/jdt13/.gem/ruby/2.2.0/bin/:

If you see anything in there reporting a 0.2.X version number, then that’ll be a problem.

Just out of interest, did you set up your PATH as outlined in the docs? If so, you shouldn’t need to be in the scripts folder to run dwi2response… You can check whether your PATH is set correctly by simply typing:

$ echo $PATH
/home/jdt13/mrtrix3/release/bin:/home/jdt13/mrtrix3/scripts:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/cuda/bin:/usr/lib32/jvm/default/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl

On my system, you can see both the MRtrix3 bin and scripts folders are listed first, which will guarantee that these commands will always by invoked preferentially, even if multiple versions of MRtrix are installed.

malav · June 14, 2016, 12:27pm

HI @jdtournier,

I am really sorry that I did not took necessity steps while reinstalling.
Now as you suggested, the result seem as below.

== mrview 0.3.14-671-g52f87d6d ==
64 bit release version, built Jun 14 2016, using Eigen 3.2.0
Author(s): J-Donald Tournier (d.tournier@brain.org.au), Dave Raffelt (david.raffelt@florey.edu.au) and Robert E. Smith (r.smith@brain.org.au)
Copyright (c) 2008-2016 the MRtrix3 contributors

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/

MRtrix is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

For more details, see www.mrtrix.org

And I added the path, so I dont have to go the release/bin or scripts every time to run the code.

But still the error is there as it can be seen.

mallo@fountainhead:~$ dwi2response tournier /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/dwi.mif /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/response.txt
dwi2response: 
dwi2response: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-help option) for more information.
dwi2response: 
dwi2response: Generated temporary directory: /tmp/dwi2response-tmp-ANS8FB/
Command: mrconvert /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/dwi.mif - -stride 0,0,0,1 | dwiextract - /tmp/dwi2response-tmp-ANS8FB/dwi.mif
dwiextract: [ERROR] invalid diffusion gradient table dimensions
dwi2response: [ERROR] Command failed: mrconvert /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/dwi.mif - -stride 0,0,0,1 | dwiextract - /tmp/dwi2response-tmp-ANS8FB/dwi.mif
dwi2response: Changing back to original directory (/home/mallo)
dwi2response: Deleting temporary directory /tmp/dwi2response-tmp-ANS8FB/

ALSO, I would like to mention specifically what I am trying to do here.
I had a 4D matrix, with I converted into a Nifti (.nii) file using Nibabel tool.
Then, I converted that file to .mif using mrconvert.
Now for tractography, I am trying to perform dwi2response to this dwi.mif file.

Keeping the possibilities in mind that I might have mistake during building the nii file, I checked the headers of the .nii file prepared, and also the .mif files. The header is something like this:

mrtrix image
dim: 1344,1024,50,16
vox: 1,1,1,1
layout: +0,+1,+2,+3
datatype: UInt16LE
transform: 1, 0, 0, 0
transform: 0, 1, 0, 0
transform: 0, 0, 1, 0
mrtrix_version: 0.3.14-666-gf2e9714b
file: . 216

Please let me know whether I need to add some other information for the gradient in this header too?
Thank you so much for your support.

jdtournier · June 14, 2016, 1:08pm

Good to hear that’s all that was needed. Your new error is about the lack of information about the DW gradients used during the acquisition. This is normally provided as a bvecs/bvals pair of files (the FSL format), or as a single text file with 4 columns for [ Gx Gy Gz b ] (MRtrix format). If using the MRtrix-specific .mif or .mih formats, this information will often be included in the image headers (especially if converted from DICOM, or from some other format if the -grad or -fslgrad options to mrconvert are used - see here for details). In your case, converting directly from NIfTI means the DW gradient information was presumably not also included in the conversion. You would need to mrconvert your data to .mif format using one of the gradient import options mentioned above. Alternatively, you can provide the same options to dwi2response directly to supply the DW gradient encoding information (type dwi2response tournier for the help page), which means something like this:

$ dwi2response tournier -fslgrad bvecs bvals /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/dwi.mif /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/response.txt

or

$ dwi2response tournier -grad encoding.b /media/mallo/Pandora/CRGof_Barcelona/At_the_CRG/Nifti/dwi.mif /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/response.txt

depending on the format you’re using for your DW gradient table. This of course all assumes that you have this information - you can’t really get anywhere without it…

malav · June 14, 2016, 1:25pm

hello,

I am a newbie at using this tool. thank you so much for sorting this out.
P.S Your assumption is right, so i hope I will reach where I am headed.

Thank you so much once again.

malav · June 14, 2016, 3:34pm

HI guys,

I am back with the older problem…
I created the gradient vector file with MRtrix format, and managed to get that detail in dwi.mif header file while using mrconvert (mrconvert -grad encoding.b myfile.nii dwi.mif)
I validated the header to make sure that nothing is left, and then tried to use dwi2response.
But,

mallo@fountainhead:/media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti$ dwi2response tournier dwi.mif response.txt
dwi2response: 
dwi2response: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-help option) for more information.
dwi2response: 
dwi2response: Generated temporary directory: /tmp/dwi2response-tmp-S29WN3/
Command: mrconvert /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/dwi.mif - -stride 0,0,0,1 | dwiextract - /tmp/dwi2response-tmp-S29WN3/dwi.mif
dwi2response: [ERROR] Command failed: mrconvert /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/dwi.mif - -stride 0,0,0,1 | dwiextract - /tmp/dwi2response-tmp-S29WN3/dwi.mif
dwi2response: Changing back to original directory (/media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti)
dwi2response: Deleting temporary directory /tmp/dwi2response-tmp-S29WN3/

To make sure that memory does not make the problem, I restarted the device, and when the root directory had 75% space free (with RAM 8 GB and my file 2.2 GB),

mallo@fountainhead:/media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti$ df
Filesystem     1K-blocks      Used Available Use% Mounted on
udev             4032620         4   4032616   1% /dev
tmpfs             808684      1212    807472   1% /run
/dev/sda9       19091584  13208288   4890428  73% /
none                   4         0         4   0% /sys/fs/cgroup
none                5120         0      5120   0% /run/lock
none             4043404       152   4043252   1% /run/shm
none              102400        48    102352   1% /run/user
/dev/sda10     105053344  18400088  81293744  19% /home
/dev/sda2         262144     32476    229668  13% /boot/efi
/dev/sda6      563199996 307806524 255393472  55% /media/mallo/Pandora

And when the command failed, I checked the memory again, and I was shown the result 85% memory (in root) being free. Also, please take notice of the issue this time that while putting error there is no mention of “brokenpipe line”.

So, I tried to run mrconvert independently, and here is how it went.

mallo@fountainhead:/media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti$ mrconvert /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/dwi.mif - -stride 0,0,0,1 | dwiextract - /tmp/dwi2response-tmp-S29WN3/dwi.mif
mrconvert: [  4%] copying from "/media/mal...na/At_the_CRG/Nifti/dwi.mif" to "/tmp/mrtrix-tmp-66h8Dx.mif"... dwiextract: [ERROR] no filename supplied to standard input (broken pipe?)
dwiextract: [ERROR] error opening image "-"

So, I guess I am back with error where mrconvert does not have enough arguement, (that’s what I am guessing).
Please help.

jdtournier · June 14, 2016, 4:44pm

OK, I think you’re confusing RAM (dynamic memory) with disk storage. df reports on disk storage, which is unlikely to be the problem here. Given what you’re showing, I think the command is running out of RAM. It’s certainly crashing, since it gets to 4%. It wouldn’t have got that far if the number of arguments was wrong. Maybe you can try this to see if you get any more useful info:

 $ mrconvert -debug /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/dwi.mif out.mif

And if that works, try writing to /tmp as the destination file:

 $ mrconvert -debug /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/dwi.mif /tmp/out.mif

rsmith · June 15, 2016, 4:49am

(the shell typically won’t report a segfault if the command is not last in the pipeline)

The scripts no longer use a single shell to execute lines that include piping; it’s done with subprocess. So all stages should be equally capable of writing to stderr - unless I’ve buggered something up.

OK, it looks like your mrview command is from the previous version of MRtrix (0.2.11).

I suspect you’re also not using the most up-to-date MRtrix3 code; the scripts will now ensure that the MRtrix3 command being executed comes from the same location as the script library being used. So I would suggest updating your installation in addition to fixing the PATH issue.

Your new error is about the lack of information about the DW gradients used during the acquisition.

This is probably worthy of its own page in the “Getting Started” section of the documentation. I’ll make a GitHub issue.

mallo@fountainhead:/media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti$ mrconvert /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/dwi.mif - -stride 0,0,0,1 | dwiextract - /tmp/dwi2response-tmp-S29WN3/dwi.mif

The issue with this test is that any issue being encountered by mrconvert may be masked by the subsequent piping to dwiextract, just as was occurring within the script. @jdtournier’s suggestion will not only remove that confound (and therefore hopefully give us more information), but the -debug option will also produce more information from the command for us to see. (Just so you know what’s going on, rather than copying and pasting commands )

malav · June 15, 2016, 9:16am

Hi @rsmith and @jdtournier,

I now am using the latest version of the MRtrix as it shows by Git.

mallo@fountainhead:~$ mrview -version
== mrview 0.3.14-671-g52f87d6d ==
64 bit release version, built Jun 14 2016, using Eigen 3.2.0
Author(s): J-Donald Tournier (d.tournier@brain.org.au), Dave Raffelt (david.raffelt@florey.edu.au) and Robert E. Smith (r.smith@brain.org.au)
Copyright (c) 2008-2016 the MRtrix3 contributors

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/

MRtrix is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

For more details, see www.mrtrix.org
mallo@fountainhead:~$ cd /home/mallo/mrtrix3/
mallo@fountainhead:~/mrtrix3$ git pull
Already up-to-date.

Thanks for suggestion of running the mrconvert with debug option to analyse the process.
I did it, trying to store the copied file first in the native folder, and then in the /tmp folder. It ran perfectly for both the cases.

mallo@fountainhead:/media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti$ mrconvert -debug dwi.mif out.nii
mrconvert: [DEBUG] No config file found at "/etc/mrtrix.conf"
mrconvert: [DEBUG] No config file found at ".mrtrix.conf"
mrconvert: [INFO] opening image "dwi.mif"...
mrconvert: [DEBUG] reading key/value file "dwi.mif"...
mrconvert: [DEBUG] sanitising image information...
mrconvert: [DEBUG] memory-mapping file "dwi.mif"...
mrconvert: [DEBUG] file "dwi.mif" mapped at 0x7fedbdaaf000, size 2202009600 (read-only)
mrconvert: [DEBUG] image "dwi.mif" loaded
mrconvert: [DEBUG] image "dwi.mif" initialised with strides = [ 1 1344 1376256 68812800 ], start = 0, using indirect IO
mrconvert: [INFO] creating image "out.nii"...
mrconvert: [DEBUG] sanitising image information...
mrconvert: [DEBUG] resizing file "out.nii" to 2202009952...
mrconvert: [DEBUG] sanitising image information...
mrconvert: [DEBUG] memory-mapping file "out.nii"...
mrconvert: [DEBUG] "out.nii" appears to reside on a networked filesystem - using delayed write-back
mrconvert: [DEBUG] file "out.nii" held in RAM at 0x7fed3a6ae010, size 2202009600
mrconvert: [DEBUG] image "out.nii" loaded
mrconvert: [DEBUG] image "out.nii" initialised with strides = [ 1 1344 1376256 68812800 ], start = 0, using indirect IO
mrconvert: [DEBUG] initialising threads...
mrconvert: [DEBUG] launching 8 threads "loop threads"...
mrconvert: [DEBUG] waiting for completion of threads "loop threads"...
mrconvert: [100%] copying from "dwi.mif" to "out.nii"... 
mrconvert: [DEBUG] threads "loop threads" completed OK
mrconvert: [100%] copying from "dwi.mif" to "out.nii"
mrconvert: [INFO] writing back contents of mapped file "out.nii"...
mrconvert: [DEBUG] image "out.nii" unloaded
mrconvert: [DEBUG] unmapping file "dwi.mif"
mrconvert: [DEBUG] image "dwi.mif" unloaded

mallo@fountainhead:/media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti$ mrconvert -debug -force dwi.mif /tmp/out.mif
mrconvert: [WARNING] existing output files will be overwritten
mrconvert: [DEBUG] No config file found at "/etc/mrtrix.conf"
mrconvert: [DEBUG] No config file found at ".mrtrix.conf"
mrconvert: [INFO] opening image "dwi.mif"...
mrconvert: [DEBUG] reading key/value file "dwi.mif"...
mrconvert: [DEBUG] sanitising image information...
mrconvert: [DEBUG] memory-mapping file "dwi.mif"...
mrconvert: [DEBUG] file "dwi.mif" mapped at 0x7f1d80c8f000, size 2202009600 (read-only)
mrconvert: [DEBUG] image "dwi.mif" loaded
mrconvert: [DEBUG] image "dwi.mif" initialised with strides = [ 1 1344 1376256 68812800 ], start = 0, using indirect IO
mrconvert: [INFO] creating image "/tmp/out.mif"...
mrconvert: [DEBUG] sanitising image information...
mrconvert: [DEBUG] resizing file "/tmp/out.mif" to 2202010320...
mrconvert: [DEBUG] sanitising image information...
mrconvert: [DEBUG] memory-mapping file "/tmp/out.mif"...
mrconvert: [DEBUG] file "/tmp/out.mif" mapped at 0x7f1cfd88e000, size 2202009600 (read-write)
mrconvert: [DEBUG] image "/tmp/out.mif" loaded
mrconvert: [DEBUG] image "/tmp/out.mif" initialised with strides = [ 1 1344 1376256 68812800 ], start = 0, using indirect IO
mrconvert: [DEBUG] initialising threads...
mrconvert: [DEBUG] launching 8 threads "loop threads"...
mrconvert: [  0%] copying from "dwi.mif" to "/tmp/out.mif"... 
mrconvert: [DEBUG] waiting for completion of threads "loop threads"...
mrconvert: [100%] copying from "dwi.mif" to "/tmp/out.mif"... 
mrconvert: [DEBUG] threads "loop threads" completed OK
mrconvert: [100%] copying from "dwi.mif" to "/tmp/out.mif"
mrconvert: [DEBUG] unmapping file "/tmp/out.mif"
mrconvert: [DEBUG] image "/tmp/out.mif" unloaded
mrconvert: [DEBUG] unmapping file "dwi.mif"
mrconvert: [DEBUG] image "dwi.mif" unloaded

So I tried to run the command with pipeline of mrconvert with dwiextract, and that’s when the problem arise, stating “error opening image “-””. There was no mention of anything about memory on terminal.

mallo@fountainhead:/media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti$ mrconvert -debug dwi.mif - -stride 0,0,0,1 | dwiextract - /tmp/dwi2response-tmp-S29WN3/dwi.mif
mrconvert: [DEBUG] No config file found at "/etc/mrtrix.conf"
mrconvert: [DEBUG] No config file found at ".mrtrix.conf"
mrconvert: [INFO] opening image "dwi.mif"...
mrconvert: [DEBUG] reading key/value file "dwi.mif"...
mrconvert: [DEBUG] sanitising image information...
mrconvert: [DEBUG] memory-mapping file "dwi.mif"...
mrconvert: [DEBUG] file "dwi.mif" mapped at 0x7fac50eb6000, size 2202009600 (read-only)
mrconvert: [DEBUG] image "dwi.mif" loaded
mrconvert: [DEBUG] image "dwi.mif" initialised with strides = [ 1 1344 1376256 68812800 ], start = 0, using indirect IO
mrconvert: [INFO] creating image "-"...
mrconvert: [DEBUG] sanitising image information...
mrconvert: [DEBUG] creating temporary file of size 0
mrconvert: [DEBUG] resizing file "/tmp/mrtrix-tmp-8GWXpO.mif" to 2202010320...
mrconvert: [DEBUG] sanitising image information...
mrconvert: [DEBUG] mapping piped image "/tmp/mrtrix-tmp-8GWXpO.mif"...
mrconvert: [DEBUG] memory-mapping file "/tmp/mrtrix-tmp-8GWXpO.mif"...
mrconvert: [DEBUG] file "/tmp/mrtrix-tmp-8GWXpO.mif" mapped at 0x7fabcdab5000, size 2202009600 (read-write)
mrconvert: [DEBUG] image "/tmp/mrtrix-tmp-8GWXpO.mif" loaded
mrconvert: [DEBUG] image "/tmp/mrtrix-tmp-8GWXpO.mif" initialised with strides = [ 16 21504 22020096 1 ], start = 0, using indirect IO
mrconvert: [DEBUG] initialising threads...
mrconvert: [DEBUG] launching 8 threads "loop threads"...
mrconvert: [  0%] copying from "dwi.mif" to "/tmp/mrtrix-tmp-8GWXpO.mif"... 
mrconvert: [DEBUG] waiting for completion of threads "loop threads"...
mrconvert: [  2%] copying from "dwi.mif" to "/tmp/mrtrix-tmp-8GWXpO.mif"... dwiextract: [ERROR] no filename supplied to standard input (broken pipe?)
dwiextract: [ERROR] error opening image "-"

Though I would like to mention that at the same time I got an alert stating that my device’s root has space left of only about 20 KB. But I don’t know whether this indicates the RAM or not as you @jdtournier mentioned earlier.

So, I guess this is how much I could get closer to the actual issu.
Any suggestion?

Thank you.

rsmith · June 15, 2016, 9:45am

Though I would like to mention that at the same time I got an alert stating that my device’s root has space left of only about 20 KB

This is absolutely going to be a problem, not just for MRtrix3 but the system in general. You really need to free up some space.

For MRtrix3, it’s slightly unusual that your test of:

mrconvert -debug -force dwi.mif /tmp/out.mif

succeeded, but the subsequent:

mrconvert -debug dwi.mif - -stride 0,0,0,1 | dwiextract - /tmp/dwi2response-tmp-S29WN3/dwi.mif

failed. Though it’s possible that this was purely due to chance / running out of hard drive space at an inopportune time. Or maybe an image with that name already existed, so you weren’t requesting any more storage.

If you look at the debugging output in the latter case, you can see lines such as this:

mrconvert: [INFO] creating image "-"...
mrconvert: [DEBUG] sanitising image information...
mrconvert: [DEBUG] creating temporary file of size 0
mrconvert: [DEBUG] resizing file "/tmp/mrtrix-tmp-8GWXpO.mif" to 2202010320...

What the use of piping (the ‘-’ symbols) here is actually doing is:

mrconvert sees the ‘-’ as its output, creates an image within /tmp/, writes its contents, and then writes the name of that image to the command line.
dwiextract sees the ‘-’ as its input, reads the name of the image from the command line (so you never see it), processes the image data, then deletes the file when it’s done.

So since you don’t have a separate file system for /tmp/ (it doesn’t show up in the output from the df command), your root filesystem needs to have enough free space in order to store this temporary file. Which is 2GB in size. And 2GB > 20KB

For the sake of the argument, an alternative way to solve such a problem is to move the location of the directory used for temporary files. This needs to be done in the MRtrix3 configuration file. If you were to move the location of this temporary directory to your home partition, you would probably find that the piping command would work. (Note that using the -tempdir option for the Python scripts does not influence the location for temporary piped files!)

But like I said, that’s for the sake of the argument. You really should clear some space on your root partition.

jdtournier · June 15, 2016, 9:53am

@rsmith is spot on here: your /tmp is not resident in RAM, as it would be in most modern Linux distributions (using tmpfs). It’s also not on a dedicated partition, residing on your root / filesystem. Letting your system run out of space on there is a Bad Idea - especially since you also don’t have a dedicated /var partition, which is where a lot of the dynamic stuff is written to (logs, print spooler, etc). Things will start going wrong if you don’t clear some space soon.

The simplest way to clear space is probably to remove any temporary files left behind by crashed MRtrix commands - of which you’ve had quite a few recently. Do this using:

$ rm /tmp/mrtrix-tmp-*

These files would normally be cleared up by the next command in the pipeline, but if the previous command crashes, the subsequent one won’t even get to know which file it’s supposed to delete… So it’s good practice to check that folder once in a while and clear it if you’ve had a crashes…

malav · June 19, 2016, 5:37pm

Hi,
so keeping in mind that memory issues don’t occur anymore, I reduced the amount of data size, and performed the dwi2response funtion. Fortunately, both the mrconvert and dwiextract woked properly.
Thank you so much for your help.

But still one more issue showed up. I ran the code and waited for the output, but I think it got stuck at the function fod2fixel. I waited for about 2 hours for the code to get finished, and I did it twice, (while keeping an eye on the memory, making sure that the device doesn’t run out of memory). Do I need to put something more information in my command? Please help me.

mallo@fountainhead:/media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti$ dwi2response tournier -grad encoding.b -mask small_mask.mif small.mif response.txt
dwi2response: 
dwi2response: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-help option) for more information.
dwi2response: 
dwi2response: Generated temporary directory: /tmp/dwi2response-tmp-R0FC4I/
Command: mrconvert /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/small.mif - -stride 0,0,0,1 -grad /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/encoding.b | dwiextract - /tmp/dwi2response-tmp-R0FC4I/dwi.mif
Command: mrconvert /media/mallo/Pandora/CRG_Barcelona/At_the_CRG/Nifti/small_mask.mif /tmp/dwi2response-tmp-R0FC4I/mask.mif -datatype bit
dwi2response: Changing to temporary directory (/tmp/dwi2response-tmp-R0FC4I/)
Command: amp2sh dwi.mif dwiSH.mif
Command: dwi2fod csd dwi.mif init_RF.txt iter0_FOD.mif -mask mask.mif -lmax 4
Command: fod2fixel iter0_FOD.mif -peak iter0_peaks.msf -mask mask.mif -fmls_no_thresholds

Thank you.

rsmith · June 20, 2016, 2:02am

I’d expect fod2fixel to run a little slower than it used to, since we increased the FOD sampling density; but I wouldn’t expect it to take 2 hours . There’s not a whole lot we can tell from a silent frozen command; the simplest thing you can do is go into the temporary directory generated by the script, and execute that command manually but include the -debug option.

malav · June 20, 2016, 9:07am

Hi @rsmith,

Thank you for your suggestion for analysing the problem while running the code.
I tried to run the fod2fixel command with -debug, and result was as following.

mallo@fountainhead:/tmp/dwi2response-tmp-2EL3TQ$ fod2fixel -force -debug iter0_FOD.mif -peak iter0_peaks.msf -mask mask.mif -fmls_no_thresholds

fod2fixel: [WARNING] existing output files will be overwritten
fod2fixel: [DEBUG] No config file found at "/etc/mrtrix.conf"
fod2fixel: [DEBUG] No config file found at ".mrtrix.conf"
fod2fixel: [INFO] opening image "iter0_FOD.mif"...
fod2fixel: [DEBUG] reading key/value file "iter0_FOD.mif"...
fod2fixel: [DEBUG] sanitising image information...
fod2fixel: [DEBUG] memory-mapping file "iter0_FOD.mif"...
fod2fixel: [DEBUG] file "iter0_FOD.mif" mapped at 0x7f4a9f0dc000, size 1303506000 (read-only)
fod2fixel: [DEBUG] image "iter0_FOD.mif" loaded
fod2fixel: [DEBUG] image "iter0_FOD.mif" initialised with strides = [ 15 10755 6517530 1 ], start = 0, using direct IO
fod2fixel: [INFO] creating image "iter0_peaks.msf"...
fod2fixel: [DEBUG] sanitising image information...
fod2fixel: [DEBUG] creating empty file "iter0_peaks.msf"
fod2fixel: [DEBUG] resizing file "iter0_peaks.msf" to 173801936
fod2fixel: [DEBUG] sanitising image information...
fod2fixel: [DEBUG] memory-mapping file "iter0_peaks.msf"...
fod2fixel: [DEBUG] file "iter0_peaks.msf" mapped at 0x7f4a94b1b000, size 173800800 (read-write)
fod2fixel: [DEBUG] Initialising output sparse data file iter0_peaks.msf: new file size 190579152 (16777216 of which is initial sparse data buffer)
fod2fixel: [DEBUG] resizing file "iter0_peaks.msf" to 190579152
fod2fixel: [DEBUG] memory-mapping file "iter0_peaks.msf"...
fod2fixel: [DEBUG] file "iter0_peaks.msf" mapped at 0x7f4a8955a000, size 16777216 (read-write)
fod2fixel: [DEBUG] image "iter0_peaks.msf" loaded
fod2fixel: [DEBUG] image "iter0_peaks.msf" initialised with strides = [ 1 717 434502 ], start = 0, using direct IO
fod2fixel: [DEBUG] Sparse image verified for accessing iter0_peaks.msf using type N2MR6Sparse11FixelMetricE
fod2fixel: [INFO] opening image "mask.mif"...
fod2fixel: [DEBUG] reading key/value file "mask.mif"...
fod2fixel: [DEBUG] sanitising image information...
fod2fixel: [DEBUG] memory-mapping file "mask.mif"...
fod2fixel: [DEBUG] file "mask.mif" mapped at 0x7f4a892c2000, size 2715638 (read-only)
fod2fixel: [DEBUG] image "mask.mif" loaded
fod2fixel: [DEBUG] image "mask.mif" initialised with strides = [ 1 717 434502 ], start = 0, using indirect IO
fod2fixel: [DEBUG] initialising threads...
fod2fixel: [DEBUG] launching thread "source"...
fod2fixel: [DEBUG] launching 8 threads "pipe"...
fod2fixel: [  0%] segmenting FODs... 
fod2fixel: [DEBUG] launching thread "sink"...
fod2fixel: [DEBUG] waiting for completion of thread "source"...
fod2fixel: [  4%] segmenting FODs... 
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!
fod2fixel: [DEBUG] failed to find SH peak!

I could not get the original source code for this command. So did not understand the reason for "failed to find SH peak!"
Will you please explain to me? (and resolve too if possible)

Thank you.

malav · June 22, 2016, 8:25am

Hello guys,

It’s a bit urgent. It would be appreiciated if you can find solution to this problem as soon as possible.

Thank you.

rsmith · June 22, 2016, 8:56am

The ‘failed to find SH peak!’ message is reasonably common, particularly since the -fmls_no_thresholds option is used. It’s trying to perform a Newton optimisation on the sphere to precisely locate the peak of each FOD lobe from the initial estimate provided by the fast-marching level-set (FMLS) FOD segmentation (Appendix 2 of this paper; but with small noisy lobes, that optimisation can fail if the FOD peak or gradients with respect to polar angles are ill-defined. So those messages doesn’t tell me a whole lot.

(The error originates from here in the code if you’re curious)

The command is getting to 4%, but this might just be due to successfully processing empty space around the brain. And I don’t see anything wrong with the rest of the log. So if it’s genuinely hanging for no good reason, I might need you to send me the files so that I can reproduce the fault and then dig into the code internals. You can send them to the email I have associated with this account.