Tckgen/mrview Segmentation fault: Invalid memory access

tractography

#1

Hello,

I think i’m being really dense and would appreciate any advise. I have seen similar posts, but have not quite been able to resolve issues running tckgen. Initially I got Segmentation fault: Invalid memory access when attempting to run tckgen. I tried various submission iterations to overcome this. I have also cleared up my storage space so that shouldn’t be an issue, so I am kind of at a loss why it isnt working as expected.

%%shell

projectDir=$WORK/connectomeProject
DATADIR=$projectDir/data ; echo "dataDir is : ${DATADIR}"  
module load mrtrix

echo -n "" > ${projectDir}/commandFiles/tractGenCommands.txt    

for subject in `ls $DATADIR`;
    do
        if [ -f ${DATADIR}/${subject}/5TT_lesion.mif ];
      then
        echo " module load \"mrtrix\"; tckgen ${DATADIR}/${subject}/WM_FODs.mif ${DATADIR}/${subject}/100M.tck -act ${DATADIR}/${subject}/5TT_lesion.mif -backtrack -crop_at_gmwmi -seed_dynamic ${DATADIR}/${subject}/WM_FODs.mif -maxlength 250 -select 100M -cutoff 0.06 -force; tcksift ${DATADIR}/${subject}/100M.tck ${DATADIR}/${subject}/WM_FODs.mif ${DATADIR}/${subject}/10M_SIFT.tck -act ${DATADIR}/${subject}/5TT_lesion.mif -term_number 10M"  >> ${projectDir}/commandFiles/tractGenCommands.txt
      else
        echo " module load \"mrtrix\"; tckgen ${DATADIR}/${subject}/WM_FODs.mif ${DATADIR}/${subject}/100M.tck -act ${DATADIR}/${subject}/5TT.mif -backtrack -crop_at_gmwmi -seed_dynamic ${DATADIR}/${subject}/WM_FODs.mif -maxlength 250 -select 100M -cutoff 0.06 -force; tcksift ${DATADIR}/${subject}/100M.tck ${DATADIR}/${subject}/WM_FODs.mif ${DATADIR}/${subject}/10M_SIFT.tck -act ${DATADIR}/${subject}/5TT.mif -term_number 10M"  >> ${projectDir}/commandFiles/tractGenCommands.txt
    fi
done
      
    # Run array job
    $HOME/c3nl_tools/hpcRunArrayJob.sh ${projectDir}/commandFiles/tractGenCommands.txt 12:00:00 48 124Gb

** Edit ** I have changed the above to generate 10M streamlines in favour of SIFT2

All the best,
Niall


#2

Dear Neuroenthusiasts,

Around half my sample have completed without error, but there seems to be certain participants who are just being uncompliant. I tried re-running previous steps and the RGB, WM_FOD look fine. I have included the debug output if anyone has any insight.

tckgen -version is 64 bit release version, built Dec 20 2017, using Eigen 3.3.4

(py27) -bash-4.2$ 
(py27) -bash-4.2$ tckgen -nthreads 0 WM_FODs.mif 10M.tck -act 5TT.mif -backtrack -crop_at_gmwmi -seed_dynamic WM_FODs.mif -maxlength 250 -select 100 -cutoff 0.06 -force -debug
tckgen: [WARNING] existing output files will be overwritten
tckgen: [DEBUG] No config file found at "/etc/mrtrix.conf"
tckgen: [DEBUG] No config file found at "/home/nbourke/.mrtrix.conf"
tckgen: [INFO] opening image "WM_FODs.mif"...
tckgen: [DEBUG] reading key/value file "WM_FODs.mif"...
tckgen: [DEBUG] sanitising image information...
tckgen: [INFO] image "WM_FODs.mif" opened with dimensions 128x128x66x45, voxel spacing 2x2x2x1, datatype Float32LE
tckgen: [DEBUG] memory-mapping file "WM_FODs.mif"...
tckgen: [DEBUG] file "WM_FODs.mif" mapped at 0x2b87d07ed000, size 194641920 (read-only)
tckgen: [DEBUG] image "WM_FODs.mif" loaded
tckgen: [DEBUG] image "WM_FODs.mif" initialised with strides = [ -45 5760 737280 1 ], start = 5715, using direct IO
tckgen: [DEBUG] sanitising image information...
tckgen: [DEBUG] allocating scratch buffer for image "fixel map voxels"...
tckgen: [DEBUG] image "fixel map voxels" loaded
tckgen: [DEBUG] image "fixel map voxels" initialised with strides = [ -1 128 16384 ], start = 127, using direct IO
tckgen: [DEBUG] sanitising image information...
tckgen: [DEBUG] allocating scratch buffer for image "SIFT model processing mask"...
tckgen: [DEBUG] image "SIFT model processing mask" loaded
tckgen: [DEBUG] image "SIFT model processing mask" initialised with strides = [ -1 128 16384 ], start = 127, using direct IO
tckgen: [INFO] opening image "5TT.mif"...
tckgen: [DEBUG] reading key/value file "5TT.mif"...
tckgen: [DEBUG] sanitising image information...
tckgen: [INFO] image "5TT.mif" opened with dimensions 128x128x66x5, voxel spacing 2x2x2xnan, datatype Float32LE
tckgen: [DEBUG] memory-mapping file "5TT.mif"...
tckgen: [DEBUG] file "5TT.mif" mapped at 0x2b87dcdf1000, size 21626880 (read-only)
tckgen: [DEBUG] image "5TT.mif" loaded
tckgen: [DEBUG] image "5TT.mif" initialised with strides = [ 5 640 81920 1 ], start = 0, using direct IO
tckgen: [DEBUG] sanitising image information...
tckgen: [DEBUG] allocating scratch buffer for image "5TT scratch buffer"...
tckgen: [DEBUG] image "5TT scratch buffer" loaded
tckgen: [DEBUG] image "5TT scratch buffer" initialised with strides = [ -5 640 81920 1 ], start = 635, using direct IO
tckgen: [INFO] 5TT image dimensions match fixel image - importing directly
tckgen: [DEBUG] unmapping file "5TT.mif"
tckgen: [DEBUG] image "5TT.mif" unloaded
tckgen: [INFO] opening image "5TT.mif"...
tckgen: [DEBUG] reading key/value file "5TT.mif"...
tckgen: [DEBUG] sanitising image information...
tckgen: [INFO] image "5TT.mif" opened with dimensions 128x128x66x5, voxel spacing 2x2x2xnan, datatype Float32LE
tckgen: [DEBUG] memory-mapping file "5TT.mif"...
tckgen: [DEBUG] file "5TT.mif" mapped at 0x2b87df733000, size 21626880 (read-only)
tckgen: [DEBUG] image "5TT.mif" loaded
tckgen: [DEBUG] image "5TT.mif" initialised with strides = [ 5 640 81920 1 ], start = 0, using direct IO
tckgen: [100%] segmenting FODs
tckgen: [INFO] opening image "WM_FODs.mif"...
tckgen: [DEBUG] reading key/value file "WM_FODs.mif"...
tckgen: [DEBUG] sanitising image information...
tckgen: [INFO] image "WM_FODs.mif" opened with dimensions 128x128x66x45, voxel spacing 2x2x2x1, datatype Float32LE
tckgen: [DEBUG] memory-mapping file "WM_FODs.mif"...
tckgen: [DEBUG] file "WM_FODs.mif" mapped at 0x2b87e0bd4000, size 194641920 (read-only)
tckgen: [DEBUG] image "WM_FODs.mif" loaded
tckgen: [DEBUG] image "WM_FODs.mif" initialised with strides = [ -45 5760 737280 1 ], start = 5715, using direct IO
tckgen: [INFO] opening image "5TT.mif"...
tckgen: [DEBUG] reading key/value file "5TT.mif"...
tckgen: [DEBUG] sanitising image information...
tckgen: [INFO] image "5TT.mif" opened with dimensions 128x128x66x5, voxel spacing 2x2x2xnan, datatype Float32LE
tckgen: [DEBUG] memory-mapping file "5TT.mif"...
tckgen: [DEBUG] file "5TT.mif" mapped at 0x2b87ec576000, size 21626880 (read-only)
tckgen: [DEBUG] image "5TT.mif" loaded
tckgen: [DEBUG] image "5TT.mif" initialised with strides = [ 5 640 81920 1 ], start = 0, using direct IO
tckgen: [INFO] step size = 1 mm
tckgen: [INFO] maximum deviation angle = 45 deg
tckgen: [INFO] minimum radius of curvature = 1.9999999443449312 mm
tckgen: [INFO] iFOD2 internal step size = 0.333333343 mm
tckgen: [DEBUG] creating empty file "10M.tck"
tckgen: [INFO] rejection sampling will use 7 directions with a ratio of 2.15373945 (predicted number of samples per step = 12.9595909)

tckgen: [SYSTEM FATAL CODE: SIGSEGV (11)] Segmentation fault: Invalid memory access
(py27) -bash-4.2$ 

I can successfully run it on my local computer
64 bit release version, built Nov 10 2016, using Eigen 3.2.10

However this kind of defeats the purpose of trying to put the whole pipeline together in a Jupyter notebook to run on our cluster :). Please tell me if I am just being dense and cant see something obvious.

Best,


#3

I’d love to… :wink: – but I’m sure I can in this instance.

Given that the same command works fine for some datasets but not others, it’s definitely worth having a good look at the command’s inputs to verify that there is nothing immediately erroneous about them. Maybe @rsmith will be able to give you some pointers as to what to look out for.

But regardless, we view any unexpected crash like this as a bug – we should check that all our assumptions are met and provide an informative error message if there is an issue with the data. But we need to figure out what’s causing the issue first, and I guess this is going to take some debugging… Can you try prefixing your command with catchsegv (I only just found out about this gem), i.e.:

catchsegv tckgen -nthreads 0 WM_FODs.mif 10M.tck -act 5TT.mif ...

See what that reports…

If you’re feeling adventurous, you could try compiling a debug build, and run it again with the catchgsegv prefix. That will run much slower, but give us a much more precise location for where the problem occurs.

And if you’re feeling very patient, you could also try running the debug version with a valgrind prefix (if it’s installed). It’ll run very slow, but will hopefully tell us straight away if there are memory handling bugs in our code…