I recently installed MRtrix on two Kubuntu OS computers (not that it matters, one laptop and one desktop). On several DWI directories I receive the error [SYSTEM FATAL CODE: SIGSEGV (11)] Segmentation fault: Invalid memory access.
Error occurs after dwidenoise: [1%] running MP-PCA denoising
However, if I run the dwidenoise command on a Mac (10.13), that I installed MRtrix on about 10 months ago, things run smoothly.
Any ideas?
I attempted to solve the problem by both updating MRtrix to the latest version and reverting to a previous version (3.0_RC2). Unfortunately, neither worked.
I think the most likely explanation (other than a bug in dwidenoise itself), might be an incompatibility with the specific versions of the libraries used on these systems. A similar issue was reported here.
Can you post the contents of your config file, and the output of ldd $(dwidenoise)? That might give us a clue.
Otherwise, we’ll need to investigate deeper, which will involve compiling a debug version and running within the debugger…
config file output:
#!/usr/bin/python
#
# autogenerated by MRtrix configure script
#
# configure output:
#
# MRtrix build type requested: release version
#
# Detecting OS: linux
# Looking for compiler [clang++]: clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
# Checking for C++11 compliance: ok
# Checking shared library generation: ok
# Detecting pointer size: 64 bit
# Detecting byte order: little-endian
# Checking for variable-length array support: ok
# Checking for non-POD variable-length array support: ok
# Checking for ::max_align_t: 16 bytes
# Checking for std::max_align_t: 16 bytes
# Checking for Eigen3 library: 3.3.4
# Checking for zlib compression library: 1.2.11
# Checking for "JSON for Modern C++" requirements: ok
# Checking for TIFF library: LIBTIFF, Version 4.0.9
# Checking for FFTW library: fftw-3.3.7-sse2-avx
# Checking for Qt moc: moc (version 5.9.5)
# Checking for Qt qmake: qmake (version 5.9.5)
# Checking for Qt rcc: rcc (version 5.9.5)
# Checking for Qt: 5.9.5
PATH = r'/home/jamie/bin/ants/bin:/mnt/Rsrch4/Documents/DTI Portfolio/shell-scripts:/usr/local/itksnap-3.8.0-beta-20181028-Linux-gcc64/bin:/usr/local/fsl/bin:/home/jamie/Downloads/mrtrix3/bin:/usr/local/fsl/bin:/home/jamie/.local/bin:/home/jamie/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin'
obj_suffix = '.o'
exe_suffix = ''
lib_prefix = 'lib'
lib_suffix = '.so'
cpp = [ 'clang++', '-c', 'CFLAGS', 'SRC', '-o', 'OBJECT' ]
cpp_flags = [ '-std=c++11', '-DMRTRIX_BUILD_TYPE="release version"', '-pthread', '-fPIC', '-DMRTRIX_WORD64', '-DMRTRIX_TIFF_SUPPORT', '-isystem', '/usr/include/x86_64-linux-gnu', '-DEIGEN_FFTW_DEFAULT', '-Wall', '-O3', '-DNDEBUG' ]
ld = [ 'clang++', 'OBJECTS', 'LDFLAGS', '-o', 'EXECUTABLE' ]
ld_flags = [ '-Wl,--sort-common,--as-needed', '-pthread', '-lz', '-ltiff', '-lfftw3' ]
runpath = '-Wl,-rpath,$ORIGIN/'
ld_enabled = True
ld_lib = [ 'clang++', 'OBJECTS', 'LDLIB_FLAGS', '-o', 'LIB' ]
ld_lib_flags = [ '-shared', '-Wl,--sort-common,--as-needed', '-pthread', '-lz', '-ltiff', '-lfftw3' ]
eigen_cflags = [ '-isystem', '/usr/include/eigen3', '-DEIGEN_DONT_PARALLELIZE' ]
moc = 'moc'
rcc = 'rcc'
qt_cflags = [ '-pipe', '-std=gnu++11', '-Wall', '-W', '-D_REENTRANT', '-fPIC', '-DQT_NO_DEBUG', '-DQT_OPENGL_LIB', '-DQT_SVG_LIB', '-DQT_WIDGETS_LIB', '-DQT_GUI_LIB', '-DQT_CORE_LIB', '-isystem', '/usr/include/x86_64-linux-gnu/qt5', '-isystem', '/usr/include/x86_64-linux-gnu/qt5/QtOpenGL', '-isystem', '/usr/include/x86_64-linux-gnu/qt5/QtSvg', '-isystem', '/usr/include/x86_64-linux-gnu/qt5/QtWidgets', '-isystem', '/usr/include/x86_64-linux-gnu/qt5/QtGui', '-isystem', '/usr/include/x86_64-linux-gnu/qt5/QtCore', '-isystem', '/usr/include/libdrm', '-isystem', '/usr/lib/x86_64-linux-gnu/qt5/mkspecs/linux-g++' ]
qt_ldflags = [ '-Wl,-O1', '-lQt5OpenGL', '-lQt5Svg', '-lQt5Widgets', '-lQt5Gui', '-lQt5Core', '-lGL', '-lpthread' ]
nogui = False
ldd $(which dwidenoise) output:
linux-vdso.so.1 (0x00007ffe9ef6f000)
libmrtrix.so => /home/jamie/Downloads/mrtrix3/bin/../lib/libmrtrix.so (0x00007fbd444d5000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fbd4414c000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fbd43dae000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fbd43b96000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbd43977000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbd43586000)
/lib64/ld-linux-x86-64.so.2 (0x00007fbd4498c000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fbd43369000)
libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007fbd430f2000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fbd42ecc000)
libjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007fbd42cbe000)
libjpeg.so.8 => /usr/lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007fbd42a56000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fbd42852000)
For complete, I executed dwidenoise with the -debug flag. The output:
>> dwidenoise -mask ./DTI32_17242/dwi_mask.nii.gz ./DTI32_17242/dwi.mif ./DTI32_17242/dwi_denoised.mif -force -debug
dwidenoise: [WARNING] existing output files will be overwritten
dwidenoise: [DEBUG] No config file found at "/etc/mrtrix.conf"
dwidenoise: [DEBUG] No config file found at "/home/jamie/.mrtrix.conf"
dwidenoise: [INFO] opening image "./DTI32_17242/dwi.mif"...
dwidenoise: [DEBUG] reading key/value file "./DTI32_17242/dwi.mif"...
dwidenoise: [DEBUG] sanitising image information...
dwidenoise: [INFO] image "./DTI32_17242/dwi.mif" opened with dimensions 256x256x57x33, voxel spacing 0.9375x0.9375x2.6000000000000001xnan, datatype UInt16LE
dwidenoise: [DEBUG] memory-mapping file "./DTI32_17242/dwi.mif"...
dwidenoise: [DEBUG] file "./DTI32_17242/dwi.mif" mapped at 0x7f3ea2f38000, size 246546432 (read-only)
dwidenoise: [DEBUG] image "./DTI32_17242/dwi.mif" loaded
dwidenoise: [DEBUG] image "./DTI32_17242/dwi.mif" initialised with strides = [ -1 -256 65536 3735552 ], start = 65535, using indirect IO
dwidenoise: [DEBUG] initialising threads...
dwidenoise: [DEBUG] launching 4 threads "loop threads"...
dwidenoise: [ 0%] preloading data for "./DTI32_17242/dwi.mif"...
dwidenoise: [DEBUG] waiting for completion of threads "loop threads"...
dwidenoise: [100%] preloading data for "./DTI32_17242/dwi.mif"...
dwidenoise: [DEBUG] threads "loop threads" completed OK
dwidenoise: [100%] preloading data for "./DTI32_17242/dwi.mif"
dwidenoise: [DEBUG] image "./DTI32_17242/dwi.mif" initialised with strides = [ -33 -8448 2162688 1 ], start = 2162655, using direct IO
dwidenoise: [INFO] opening image "./DTI32_17242/dwi_mask.nii.gz"...
dwidenoise: [DEBUG] transforms_match: FOV difference in scanner coordinates: 0
dwidenoise: [DEBUG] sanitising image information...
dwidenoise: [INFO] Axes and transform of image "./DTI32_17242/dwi_mask.nii.gz" altered to approximate RAS coordinate system
dwidenoise: [DEBUG] searching for suitable phase encoding data...
dwidenoise: [INFO] image "./DTI32_17242/dwi_mask.nii.gz" opened with dimensions 256x256x57x33, voxel spacing 0.9375x0.9375x2.5999999046325684x1, datatype Int16LE
dwidenoise: [DEBUG] loading image "./DTI32_17242/dwi_mask.nii.gz"...
dwidenoise: [100%] uncompressing image "./DTI32_17242/dwi_mask.nii.gz"
dwidenoise: [DEBUG] image "./DTI32_17242/dwi_mask.nii.gz" loaded
dwidenoise: [DEBUG] image "./DTI32_17242/dwi_mask.nii.gz" initialised with strides = [ -1 -256 65536 3735552 ], start = 65535, using indirect IO
dwidenoise: [INFO] creating image "./DTI32_17242/dwi_denoised.mif"...
dwidenoise: [DEBUG] sanitising image information...
dwidenoise: [DEBUG] creating empty file "./DTI32_17242/dwi_denoised.mif"
dwidenoise: [DEBUG] resizing file "./DTI32_17242/dwi_denoised.mif" to 493095824
dwidenoise: [DEBUG] sanitising image information...
dwidenoise: [INFO] image "./DTI32_17242/dwi_denoised.mif" created with dimensions 256x256x57x33, voxel spacing 0.9375x0.9375x2.6000000000000001xnan, datatype Float32LE
dwidenoise: [DEBUG] memory-mapping file "./DTI32_17242/dwi_denoised.mif"...
dwidenoise: [DEBUG] "./DTI32_17242/dwi_denoised.mif" appears to reside on a networked filesystem - using delayed write-back
dwidenoise: [DEBUG] file "./DTI32_17242/dwi_denoised.mif" held in RAM at 0x7f3e3fe9e010, size 493092864
dwidenoise: [DEBUG] image "./DTI32_17242/dwi_denoised.mif" loaded
dwidenoise: [DEBUG] image "./DTI32_17242/dwi_denoised.mif" initialised with strides = [ -33 -8448 2162688 1 ], start = 2162655, using direct IO
dwidenoise: [DEBUG] initialising threads...
dwidenoise: [DEBUG] launching 4 threads "loop threads"...
dwidenoise: [ 0%] running MP-PCA denoising...
dwidenoise: [DEBUG] waiting for completion of threads "loop threads"...
dwidenoise: [ 1%] running MP-PCA denoising...
dwidenoise: [SYSTEM FATAL CODE: SIGSEGV (11)] Segmentation fault: Invalid memory access
These lines stick out
dwidenoise: [DEBUG] No config file found at "/etc/mrtrix.conf"
dwidenoise: [DEBUG] No config file found at "/home/jamie/.mrtrix.conf"
OK, none of that seems particularly problematic. Even the ‘No config file’ lines are perfectly normal, it just means you don’t have your own config files.
Can you try running this within a debugger:
gdb --args dwidenoise -mask ./DTI32_17242/dwi_mask.nii.gz ./DTI32_17242/dwi.mif ./DTI32_17242/dwi_denoised.mif -force -debug
...
(gdb) run
...
(wait for crash)
...
(gdb) bt
and post the output here? That might give us enough info without requiring a full debug build…
Otherwise, the simplest might be for you to send me the file so I can try to replicate. Not enormously hopefully though if you say the same command completes OK on your macOS system with the same input… ?
Apologies for the delay in response, grant due Friday. Will return!
Grants, they never end! Sorry for the delay, here is the output of the debug
(gdb) run
Starting program: /home/jatlab-remote/mrtrix3/bin/dwidenoise -mask ./DTI32_17242/dwi_mask.nii.gz ./DTI32_17242/dwi.mif ./DTI32_17242/dwi_denoised.mif -force -debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
dwidenoise: [WARNING] existing output files will be overwritten
dwidenoise: [DEBUG] No config file found at "/etc/mrtrix.conf"
dwidenoise: [DEBUG] No config file found at "/home/jatlab-remote/.mrtrix.conf"
dwidenoise: [INFO] opening image "./DTI32_17242/dwi.mif"...
dwidenoise: [DEBUG] reading key/value file "./DTI32_17242/dwi.mif"...
dwidenoise: [DEBUG] sanitising image information...
dwidenoise: [INFO] image "./DTI32_17242/dwi.mif" opened with dimensions 256x256x57x33, voxel spacing 0.9375x0.9375x2.6000000000000001xnan, datatype UInt16LE
dwidenoise: [DEBUG] memory-mapping file "./DTI32_17242/dwi.mif"...
dwidenoise: [DEBUG] file "./DTI32_17242/dwi.mif" mapped at 0x7fffe717a000, size 246546432 (read-only)
dwidenoise: [DEBUG] image "./DTI32_17242/dwi.mif" loaded
dwidenoise: [DEBUG] image "./DTI32_17242/dwi.mif" initialised with strides = [ -1 -256 65536 3735552 ], start = 65535, using indirect IO
dwidenoise: [DEBUG] initialising threads...
dwidenoise: [DEBUG] launching 32 threads "loop threads"...
[New Thread 0x7fffc9b38700 (LWP 85296)]
dwidenoise: [ 0%] preloading data for "./DTI32_17242/dwi.mif"... [New Thread 0x7fffc9337700 (LWP 85297)]
[New Thread 0x7fffc8b36700 (LWP 85298)]
[New Thread 0x7fffbbfff700 (LWP 85299)]
[New Thread 0x7fffbb7fe700 (LWP 85300)]
[New Thread 0x7fffbaffd700 (LWP 85301)]
[New Thread 0x7fffba7fc700 (LWP 85302)]
[New Thread 0x7fffb9ffb700 (LWP 85303)]
[New Thread 0x7fffb97fa700 (LWP 85304)]
[New Thread 0x7fffb8ff9700 (LWP 85305)]
[New Thread 0x7fff9bfff700 (LWP 85306)]
[New Thread 0x7fff93fff700 (LWP 85307)]
[New Thread 0x7fff9b7fe700 (LWP 85308)]
[New Thread 0x7fff9affd700 (LWP 85309)]
[New Thread 0x7fff9a7fc700 (LWP 85310)]
[New Thread 0x7fff99ffb700 (LWP 85311)]
[New Thread 0x7fff997fa700 (LWP 85312)]
[New Thread 0x7fff98ff9700 (LWP 85313)]
[New Thread 0x7fff937fe700 (LWP 85314)]
[New Thread 0x7fff92ffd700 (LWP 85315)]
[New Thread 0x7fff927fc700 (LWP 85316)]
[New Thread 0x7fff91ffb700 (LWP 85317)]
[New Thread 0x7fff917fa700 (LWP 85318)]
[New Thread 0x7fff90ff9700 (LWP 85319)]
[New Thread 0x7fff5bfff700 (LWP 85320)]
[New Thread 0x7fff53fff700 (LWP 85321)]
[New Thread 0x7fff5b7fe700 (LWP 85322)]
[New Thread 0x7fff5affd700 (LWP 85323)]
[New Thread 0x7fff5a7fc700 (LWP 85324)]
[New Thread 0x7fff59ffb700 (LWP 85325)]
[New Thread 0x7fff597fa700 (LWP 85326)]
[New Thread 0x7fff58ff9700 (LWP 85327)]
dwidenoise: [DEBUG] waiting for completion of threads "loop threads"...
dwidenoise: [100%] preloading data for "./DTI32_17242/dwi.mif"... [Thread 0x7fff58ff9700 (LWP 85327) exited]
[Thread 0x7fff597fa700 (LWP 85326) exited]
[Thread 0x7fff59ffb700 (LWP 85325) exited]
[Thread 0x7fff5a7fc700 (LWP 85324) exited]
[Thread 0x7fff5affd700 (LWP 85323) exited]
[Thread 0x7fff5b7fe700 (LWP 85322) exited]
[Thread 0x7fff53fff700 (LWP 85321) exited]
[Thread 0x7fff5bfff700 (LWP 85320) exited]
[Thread 0x7fff90ff9700 (LWP 85319) exited]
[Thread 0x7fff917fa700 (LWP 85318) exited]
[Thread 0x7fff91ffb700 (LWP 85317) exited]
[Thread 0x7fff927fc700 (LWP 85316) exited]
[Thread 0x7fff92ffd700 (LWP 85315) exited]
[Thread 0x7fff937fe700 (LWP 85314) exited]
[Thread 0x7fff98ff9700 (LWP 85313) exited]
[Thread 0x7fff997fa700 (LWP 85312) exited]
[Thread 0x7fff99ffb700 (LWP 85311) exited]
[Thread 0x7fff9a7fc700 (LWP 85310) exited]
[Thread 0x7fff9affd700 (LWP 85309) exited]
[Thread 0x7fff9b7fe700 (LWP 85308) exited]
[Thread 0x7fff93fff700 (LWP 85307) exited]
[Thread 0x7fff9bfff700 (LWP 85306) exited]
[Thread 0x7fffb8ff9700 (LWP 85305) exited]
[Thread 0x7fffb97fa700 (LWP 85304) exited]
[Thread 0x7fffb9ffb700 (LWP 85303) exited]
[Thread 0x7fffba7fc700 (LWP 85302) exited]
[Thread 0x7fffbb7fe700 (LWP 85300) exited]
[Thread 0x7fffbbfff700 (LWP 85299) exited]
[Thread 0x7fffc8b36700 (LWP 85298) exited]
[Thread 0x7fffc9337700 (LWP 85297) exited]
[Thread 0x7fffc9b38700 (LWP 85296) exited]
[Thread 0x7fffbaffd700 (LWP 85301) exited]
dwidenoise: [DEBUG] threads "loop threads" completed OK
dwidenoise: [100%] preloading data for "./DTI32_17242/dwi.mif"
dwidenoise: [DEBUG] image "./DTI32_17242/dwi.mif" initialised with strides = [ -33 -8448 2162688 1 ], start = 2162655, using direct IO
dwidenoise: [INFO] opening image "./DTI32_17242/dwi_mask.nii.gz"...
dwidenoise: [DEBUG] transforms_match: FOV difference in scanner coordinates: 0
dwidenoise: [DEBUG] sanitising image information...
dwidenoise: [INFO] Axes and transform of image "./DTI32_17242/dwi_mask.nii.gz" altered to approximate RAS coordinate system
dwidenoise: [DEBUG] searching for suitable phase encoding data...
dwidenoise: [INFO] image "./DTI32_17242/dwi_mask.nii.gz" opened with dimensions 256x256x57x33, voxel spacing 0.9375x0.9375x2.5999999046325684x1, datatype Int16LE
dwidenoise: [DEBUG] loading image "./DTI32_17242/dwi_mask.nii.gz"...
dwidenoise: [100%] uncompressing image "./DTI32_17242/dwi_mask.nii.gz"
dwidenoise: [DEBUG] image "./DTI32_17242/dwi_mask.nii.gz" loaded
dwidenoise: [DEBUG] image "./DTI32_17242/dwi_mask.nii.gz" initialised with strides = [ -1 -256 65536 3735552 ], start = 65535, using indirect IO
dwidenoise: [INFO] creating image "./DTI32_17242/dwi_denoised.mif"...
dwidenoise: [DEBUG] sanitising image information...
dwidenoise: [DEBUG] creating empty file "./DTI32_17242/dwi_denoised.mif"
dwidenoise: [DEBUG] resizing file "./DTI32_17242/dwi_denoised.mif" to 493095856
dwidenoise: [DEBUG] sanitising image information...
dwidenoise: [INFO] image "./DTI32_17242/dwi_denoised.mif" created with dimensions 256x256x57x33, voxel spacing 0.9375x0.9375x2.6000000000000001xnan, datatype Float32LE
dwidenoise: [DEBUG] memory-mapping file "./DTI32_17242/dwi_denoised.mif"...
dwidenoise: [DEBUG] "./DTI32_17242/dwi_denoised.mif" appears to reside on a networked filesystem - using delayed write-back
dwidenoise: [DEBUG] file "./DTI32_17242/dwi_denoised.mif" held in RAM at 0x7fff07e9e010, size 493092864
dwidenoise: [DEBUG] image "./DTI32_17242/dwi_denoised.mif" loaded
dwidenoise: [DEBUG] image "./DTI32_17242/dwi_denoised.mif" initialised with strides = [ -33 -8448 2162688 1 ], start = 2162655, using direct IO
dwidenoise: [DEBUG] initialising threads...
dwidenoise: [DEBUG] launching 32 threads "loop threads"...
[New Thread 0x7fff58ff9700 (LWP 85330)]
dwidenoise: [ 0%] running MP-PCA denoising... [New Thread 0x7fff597fa700 (LWP 85331)]
[New Thread 0x7fff59ffb700 (LWP 85332)]
[New Thread 0x7fff5a7fc700 (LWP 85333)]
[New Thread 0x7fffc9b38700 (LWP 85334)]
[New Thread 0x7fffc9337700 (LWP 85335)]
dwidenoise: [ 1%] running MP-PCA denoising... [New Thread 0x7fffc8b36700 (LWP 85336)]
[New Thread 0x7fffbbfff700 (LWP 85337)]
dwidenoise: [ 2%] running MP-PCA denoising... [New Thread 0x7fffbb7fe700 (LWP 85338)]
Thread 36 "dwidenoise" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff59ffb700 (LWP 85332)]
0x00007ffff79c50d4 in bool MR::(anonymous namespace)::__fetch_LE<bool, short>(void const*, unsigned long, double, double) ()
from /home/jatlab-remote/mrtrix3/bin/../lib/libmrtrix.so
(gdb)
I can send you the file, let me know if and where to send it.
Can you execute “bt” within GDB? The backtrace is crucial to see exactly where in the code the error is arising from.
I updated the packages and restarted…the error went away 
I wish I had seen the bt @rsmith called attention to it! Apologies guys.
On thing to note, I was connecting via ssh to run dwidenoise with this error (on both unix and linux) - in case this comes up again for anybody.
Wait…oddly, the issue returned on a new file. One note to make on the previous file error - I had executed dwidenoise successfully (on my mac) then after the ‘dwi_denoised.nii.gz’ file had been created (and after updating + restarting) I re-ran dwidenoise on my Linux (through ssh, though I recreated the problem directly on the Linux) and wrote over the dwidenoise generated file. Here is the (gdb) bt
(gdb) run
Starting program: /home/jatlab-remote/mrtrix3/bin/dwidenoise -mask ./DTI32_15697/dwi_mask.nii.gz ./DTI32_15697/dwi.mif ./DTI32_15697/dwi_denoised.mif -force -debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
dwidenoise: [WARNING] existing output files will be overwritten
dwidenoise: [DEBUG] No config file found at "/etc/mrtrix.conf"
dwidenoise: [DEBUG] No config file found at "/home/jatlab-remote/.mrtrix.conf"
dwidenoise: [INFO] opening image "./DTI32_15697/dwi.mif"...
dwidenoise: [DEBUG] reading key/value file "./DTI32_15697/dwi.mif"...
dwidenoise: [DEBUG] sanitising image information...
dwidenoise: [INFO] image "./DTI32_15697/dwi.mif" opened with dimensions 256x256x57x33, voxel spacing 0.9375x0.9375x2.59999xnan, datatype UInt16LE
dwidenoise: [DEBUG] memory-mapping file "./DTI32_15697/dwi.mif"...
dwidenoise: [DEBUG] file "./DTI32_15697/dwi.mif" mapped at 0x7fffe717a000, size 246546432 (read-only)
dwidenoise: [DEBUG] image "./DTI32_15697/dwi.mif" loaded
dwidenoise: [DEBUG] image "./DTI32_15697/dwi.mif" initialised with strides = [ -1 -256 65536 3735552 ], start = 65535, using indirect IO
dwidenoise: [DEBUG] initialising threads...
dwidenoise: [DEBUG] launching 32 threads "loop threads"...
[New Thread 0x7fffc9b38700 (LWP 59723)]
dwidenoise: [ 0%] preloading data for "./DTI32_15697/dwi.mif"... [New Thread 0x7fffc9337700 (LWP 59724)]
[New Thread 0x7fffc8b36700 (LWP 59725)]
[New Thread 0x7fffc3fff700 (LWP 59726)]
[New Thread 0x7fffc37fe700 (LWP 59727)]
[New Thread 0x7fffc2ffd700 (LWP 59728)]
[New Thread 0x7fffc27fc700 (LWP 59729)]
[New Thread 0x7fffc1ffb700 (LWP 59730)]
[New Thread 0x7fffc17fa700 (LWP 59731)]
[New Thread 0x7fffc0ff9700 (LWP 59732)]
[New Thread 0x7fff9ffff700 (LWP 59733)]
[New Thread 0x7fff9f7fe700 (LWP 59734)]
[New Thread 0x7fff9effd700 (LWP 59735)]
[New Thread 0x7fff9e7fc700 (LWP 59736)]
[New Thread 0x7fff9dffb700 (LWP 59737)]
[New Thread 0x7fff9d7fa700 (LWP 59738)]
[New Thread 0x7fff9cff9700 (LWP 59739)]
[New Thread 0x7fff7bfff700 (LWP 59740)]
[New Thread 0x7fff7b7fe700 (LWP 59741)]
[New Thread 0x7fff7affd700 (LWP 59742)]
[New Thread 0x7fff7a7fc700 (LWP 59743)]
[New Thread 0x7fff79ffb700 (LWP 59744)]
[New Thread 0x7fff797fa700 (LWP 59745)]
[New Thread 0x7fff78ff9700 (LWP 59746)]
[New Thread 0x7fff5bfff700 (LWP 59747)]
[New Thread 0x7fff5b7fe700 (LWP 59748)]
[New Thread 0x7fff5affd700 (LWP 59749)]
[New Thread 0x7fff5a7fc700 (LWP 59750)]
[New Thread 0x7fff59ffb700 (LWP 59751)]
[New Thread 0x7fff597fa700 (LWP 59752)]
[New Thread 0x7fff58ff9700 (LWP 59753)]
[New Thread 0x7fff3bfff700 (LWP 59754)]
dwidenoise: [DEBUG] waiting for completion of threads "loop threads"...
dwidenoise: [100%] preloading data for "./DTI32_15697/dwi.mif"... [Thread 0x7fff58ff9700 (LWP 59753) exited]
[Thread 0x7fff597fa700 (LWP 59752) exited]
[Thread 0x7fff3bfff700 (LWP 59754) exited]
[Thread 0x7fff59ffb700 (LWP 59751) exited]
[Thread 0x7fff5a7fc700 (LWP 59750) exited]
[Thread 0x7fff5affd700 (LWP 59749) exited]
[Thread 0x7fff5b7fe700 (LWP 59748) exited]
[Thread 0x7fff5bfff700 (LWP 59747) exited]
[Thread 0x7fff78ff9700 (LWP 59746) exited]
[Thread 0x7fff797fa700 (LWP 59745) exited]
[Thread 0x7fff79ffb700 (LWP 59744) exited]
[Thread 0x7fff7a7fc700 (LWP 59743) exited]
[Thread 0x7fff7affd700 (LWP 59742) exited]
[Thread 0x7fff7b7fe700 (LWP 59741) exited]
[Thread 0x7fff7bfff700 (LWP 59740) exited]
[Thread 0x7fff9cff9700 (LWP 59739) exited]
[Thread 0x7fff9d7fa700 (LWP 59738) exited]
[Thread 0x7fff9dffb700 (LWP 59737) exited]
[Thread 0x7fff9e7fc700 (LWP 59736) exited]
[Thread 0x7fff9effd700 (LWP 59735) exited]
[Thread 0x7fff9f7fe700 (LWP 59734) exited]
[Thread 0x7fff9ffff700 (LWP 59733) exited]
[Thread 0x7fffc0ff9700 (LWP 59732) exited]
[Thread 0x7fffc17fa700 (LWP 59731) exited]
[Thread 0x7fffc1ffb700 (LWP 59730) exited]
dwidenoise: [DEBUG] threads "loop threads" completed OK
[Thread 0x7fffc27fc700 (LWP 59729) exited]
[Thread 0x7fffc2ffd700 (LWP 59728) exited]
dwidenoise: [100%] preloading data for "./DTI32_15697/dwi.mif"
[Thread 0x7fffc37fe700 (LWP 59727) exited]
[Thread 0x7fffc3fff700 (LWP 59726) exited]
[Thread 0x7fffc8b36700 (LWP 59725) exited]
dwidenoise: [DEBUG] image "./DTI32_15697/dwi.mif" initialised with strides = [ -33 -8448 2162688 1 ], start = 2162655, using direct IO
[Thread 0x7fffc9337700 (LWP 59724) exited]
[Thread 0x7fffc9b38700 (LWP 59723) exited]
dwidenoise: [INFO] opening image "./DTI32_15697/dwi_mask.nii.gz"...
dwidenoise: [DEBUG] transforms_match: FOV difference in scanner coordinates: 0
dwidenoise: [DEBUG] sanitising image information...
dwidenoise: [INFO] Axes and transform of image "./DTI32_15697/dwi_mask.nii.gz" altered to approximate RAS coordinate system
dwidenoise: [DEBUG] searching for suitable phase encoding data...
dwidenoise: [INFO] image "./DTI32_15697/dwi_mask.nii.gz" opened with dimensions 256x256x57x33, voxel spacing 0.9375x0.9375x2.5999898910522461x1, datatype Int16LE
dwidenoise: [DEBUG] loading image "./DTI32_15697/dwi_mask.nii.gz"...
dwidenoise: [100%] uncompressing image "./DTI32_15697/dwi_mask.nii.gz"
dwidenoise: [DEBUG] image "./DTI32_15697/dwi_mask.nii.gz" loaded
dwidenoise: [DEBUG] image "./DTI32_15697/dwi_mask.nii.gz" initialised with strides = [ -1 -256 65536 3735552 ], start = 65535, using indirect IO
dwidenoise: [INFO] creating image "./DTI32_15697/dwi_denoised.mif"...
dwidenoise: [DEBUG] sanitising image information...
dwidenoise: [DEBUG] creating empty file "./DTI32_15697/dwi_denoised.mif"
dwidenoise: [DEBUG] resizing file "./DTI32_15697/dwi_denoised.mif" to 493095848
dwidenoise: [DEBUG] sanitising image information...
dwidenoise: [INFO] image "./DTI32_15697/dwi_denoised.mif" created with dimensions 256x256x57x33, voxel spacing 0.9375x0.9375x2.59999xnan, datatype Float32LE
dwidenoise: [DEBUG] memory-mapping file "./DTI32_15697/dwi_denoised.mif"...
dwidenoise: [DEBUG] "./DTI32_15697/dwi_denoised.mif" appears to reside on a networked filesystem - using delayed write-back
dwidenoise: [DEBUG] file "./DTI32_15697/dwi_denoised.mif" held in RAM at 0x7fff07e9e010, size 493092864
dwidenoise: [DEBUG] image "./DTI32_15697/dwi_denoised.mif" loaded
dwidenoise: [DEBUG] image "./DTI32_15697/dwi_denoised.mif" initialised with strides = [ -33 -8448 2162688 1 ], start = 2162655, using direct IO
dwidenoise: [DEBUG] initialising threads...
dwidenoise: [DEBUG] launching 32 threads "loop threads"...
[New Thread 0x7fff3bfff700 (LWP 59755)]
dwidenoise: [ 0%] running MP-PCA denoising... [New Thread 0x7fff58ff9700 (LWP 59756)]
[New Thread 0x7fff597fa700 (LWP 59757)]
[New Thread 0x7fff59ffb700 (LWP 59758)]
[New Thread 0x7fffc9b38700 (LWP 59759)]
[New Thread 0x7fffc9337700 (LWP 59760)]
dwidenoise: [ 1%] running MP-PCA denoising... [New Thread 0x7fffc8b36700 (LWP 59761)]
[New Thread 0x7fffc3fff700 (LWP 59762)]
dwidenoise: [ 2%] running MP-PCA denoising... [New Thread 0x7fffc37fe700 (LWP 59763)]
[New Thread 0x7fffc2ffd700 (LWP 59764)]
Thread 36 "dwidenoise" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff597fa700 (LWP 59757)]
0x00007ffff79c50d4 in bool MR::(anonymous namespace)::__fetch_LE<bool, short>(void const*, unsigned long, double, double) () from /home/jatlab-remote/mrtrix3/bin/../lib/libmrtrix.so
(gdb) bt
#0 0x00007ffff79c50d4 in bool MR::(anonymous namespace)::__fetch_LE<bool, short>(void const*, unsigned long, double, double) () from /home/jatlab-remote/mrtrix3/bin/../lib/libmrtrix.so
#1 0x0000000000425d75 in DenoisingFunctor<MR::Image<float> >::operator()(MR::Image<float>&, MR::Image<float>&) ()
#2 0x0000000000414da2 in MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >::operator()(MR::Iterator const&) ()
#3 0x00000000004159e4 in MR::(anonymous namespace)::ThreadedLoopRunOuter<MR::LoopAlongDynamicAxesProgress>::run_outer<MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&>(MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&)::PerThread::execute() ()
#4 0x000000000041619e in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<void (MR::(anonymous namespace)::ThreadedLoopRunOuter<MR::LoopAlongDynamicAxesProgress>::run_outer<MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&>(MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&)::PerThread::*)(), MR::(anonymous namespace)::ThreadedLoopRunOuter<MR::LoopAlongDynamicAxesProgress>::run_outer<MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&>(MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&)::PerThread*> >, void> >::_M_invoke(std::_Any_data const&) ()
#5 0x000000000041dfc7 in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)
()
#6 0x00007ffff6dcf827 in __pthread_once_slow (once_control=0x65c2b8, init_routine=0x7ffff7651760 <__once_proxy>) at pthread_once.c:116
#7 0x000000000041604b in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<void (MR::(anonymous namespace)::ThreadedLoopRunOuter<MR::LoopAlongDynamicAxesProgress>::run_outer<MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&>(MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&)::PerThread::*)(), MR::(anonymous namespace)::ThreadedLoopRunOuter<MR::LoopAlongDynamicAxesProgress>::run_outer<MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&>(MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&)::PerThread*> >, void>::_Async_state_impl(std::thread::_Invoker<std::tuple<void (MR::(anonymous namespace)::ThreadedLoopRunOuter<MR::LoopAlongDynamicAxesProgress>::run_outer<MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&>(MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&)::PerThread::*)(), MR::(anonymous namespace)::ThreadedLoopRunOuter<MR::LoopAlongDynamicAxesProgress>::run_outer<MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&>(MR::(anonymous namespace)::ThreadedLoopRunInner<2, DenoisingFunctor<MR::Image<float> >, MR::Image<float>, MR::Image<float> >&)::PerThread*> >&&)::{lambda()#1}> > >::_M_run() ()
#8 0x00007ffff765257f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007ffff6dc76db in start_thread (arg=0x7fff597fa700) at pthread_create.c:463
#10 0x00007ffff6af088f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
OK, this is getting weird… Can I confirm that the same command worked ok on the same input file, on the same system, but crashed out the second time? This command is completely deterministic, the actions performed should be identical in both cases… The only things that might introduce a bit of randomness is the multi-threading. Does this still crash out if you add the -nthread 0 option?
Otherwise it’s very hard to see why this would go wrong. The crash happens as it’s fetching data from the input image, which it’s already preloaded into RAM at this point. I’d only expect this to happen if the program was fetching memory from an invalid location, and I don’t see what could cause this to happen…
One thing to try might be to use a more recent version of Eigen, but that’s a long shot…
Ignore that, you’ve reported your version of Eigen as 3.3.4, that should be recent enough…
It is weird. On the Linux, adding -nthread 0 regardless of dwi.mif's origin is successful, and unsuccessful without.
The commands I execute prior to dwidenoise are:
mrconvert "${dwiDir}/dicom" "${dwiDir}/dwi.mif"
mrconvert "${dwiDir}/dwi.mif" "${dwiDir}/temp.nii.gz"
bet2 "${dwiDir}/temp.nii.gz" "${dwiDir}/dwi" -f 0.25 -m -n
If I execute these commands on my Linux, then dwidenoise - I receive the Segmentation fault error:
dwidenoise -mask "${dwiDir}/dwi_mask.nii.gz" "${dwiDir}/dwi.mif" "${dwiDir}/dwi_denoised.mif"
dwidenoise: [100%] preloading data for "/mnt/Rsrch4/ImgData/TissueClass2018/TC01/tmp/dwi.mif"
dwidenoise: [100%] uncompressing image "/mnt/Rsrch4/ImgData/TissueClass2018/TC01/tmp/dwi_mask.nii.gz"
dwidenoise: [ 2%] running MP-PCA denoising...
dwidenoise: [SYSTEM FATAL CODE: SIGSEGV (11)] Segmentation fault: Invalid memory access
However, if I run the commands prior to dwidenoise from the Mac and execute dwidenoise on the Linux - it’s successful:
dwidenoise -mask "${dwiDir}/dwi_mask.nii.gz" "${dwiDir}/dwi.mif" "${dwiDir}/dwi_denoised.mif"
dwidenoise: [100%] preloading data for "/mnt/Rsrch4/ImgData/TissueClass2018/TC01/tmp/dwi.mif"
dwidenoise: [100%] uncompressing image "/mnt/Rsrch4/ImgData/TissueClass2018/TC01/tmp/dwi_mask.nii.gz"
dwidenoise: [100%] running MP-PCA denoising
Including -nthread 0 here, dwidenoise executes without error. Any other thoughts? It’d be fantastic to have the whole she-bang execute on Linux.
Strange… There may be issues introduced by the actual filesystem – I assume these files are held on a network drive, itself accessed using each OS’s own drivers? In which case, there may be a subtle bug on one or the other OS’s filesystem drivers.
If that’s not the case, I’d be interested to narrow down whether the issue is introduced by the mask, which is produced outside of MRtrix3. Are there any issues when running dwidenoise without the -mask option?
If not, then we need to figure out what’s different about the mask when produced on macOS vs. Linux. I note it’s a 33 volume image, when I’d expect a single volume. This shouldn’t matter though, dwidenoise will only use the first volume, but it might hint at where the issue might be.
If there is a difference, then we need to figure out what is about our handling of that image that’s causing the crash…
I’ll look into what I can find about OS’s filesystem driver repairs, updates, etc. One note: dwidenoise failed on two computers with the version of Linux I am using (Kubuntu 18.04 LTS) - which makes me think that if this were the problem it is more likely OS specific rather than a corrupt copy.
The command dwidenoise "${dwiDir}/dwi.mif" "${dwiDir}/dwi_denoised.mif" executed successfully on the problematic Linux. So I moved on to creating a mask with dwi2mask and re-ran dwidenoise as dwidenoise "${dwiDir}/dwi.mif" "${dwiDir}/dwi_denoised.mif" -mask "${dwiDir}/dwi_mask.nii.gz - another success!
Lastly, I extracted a b0 image using dwiextract then created a mask of it using bet2. Executing dwidenoise "${dwiDir}/dwi.mif" "${dwiDir}/dwi_denoised.mif" -mask "${dwiDir}/b0_bet2_mask.nii.gz" worked too.
Sorry, it’s been a while since I’ve had the chance to look into this issue. It’s looking likely that the problem might relate to a badly formed mask image – in this case derived using bet2. Any chance you could send me the problematic dwi_mask.nii.gz image for inspection…?
Yes, I can. I’ll send it along this week!
Hi all,
I wanted to report a similar experience. I had the same problem (dwidenoise: [SYSTEM FATAL CODE: SIGSEGV (11)] Segmentation fault: Invalid memory access) trying to run dwidenoise with a bet2 mask. Running dwidenoise without the mask or with a mask generated by dwi2mask fixes the problem.
Thank you for the very useful thread.
Luis
Any chance you could share the data that leads to the segfault so I can reproduce and investigate?
I am working with subjects from the HBN data set here. I ran into the above error analyzing subjects: NDARYH480GTD and NDARDJ825GBP.
I think I have isolated the problem to brain masks created with bet2 from the entire 4d data set. If I first extract the b0 volume from the dwi dataset using fslroi and run bet2 -m on that volume, dwidenoise runs fine.
Runing the following piece of script returns the error:
foreach -20 *_acq-64dir_dwi.nii.gz : \
bet2 IN UNI_dwi_brain -m ";" \
dwidenoise -mask UNI_dwi_brain_mask.nii.gz -noise UNI_noise.nii.gz IN UNI_denoised.nii.gz "&&" \
mrcalc IN UNI_denoised.nii.gz -subtract UNI_residuals.nii.gz ";" \
dwidenoise: [SYSTEM FATAL CODE: SIGSEGV (11)] Segmentation fault: Invalid memory access:
In this next script, I remove the b0 volume from the data set and run bet2 on that volume. I do not get the error here.
foreach -20 *_acq-64dir_dwi.nii.gz : \
fslroi IN UNI_dwi_b0 0 1 ";" \
bet2 UNI_dwi_b0 UNI_dwi_b0_brain -m
dwidenoise -mask UNI_dwi_b0_brain_mask.nii.gz -noise UNI_noise.nii.gz IN UNI_denoised.nii.gz "&&" \
mrcalc IN UNI_denoised.nii.gz -subtract UNI_residuals.nii.gz
Best,
Luis