Dwi2tensor crash multithreading windows

1

Hi!
My computer crashes when I run dwi2tensor unless I turn off multi-threading (-nthreads 0). It’s not a big problem and I presume this has something to do with running MRtrix3 on Windows? Anyway, I thought I’d mention it; and perhaps also that I didn’t experience this problem before updating MRtrix3 recently. I think the last version was from before summer sometime (sorry for being so unspecific).

The computer:
Windows 7 Enterprise sp1
Intel® Core™ i7-4930K CPU @ 3.40GHz
RAM 32 GB

Cheers,
Örjan

2

That’s unfortunate… Any chance you could post the contents of your release/config file? And also confirm that you’re running a recent version (i.e. what does dwi2tensor -version report?). Also, what is the full command line that causes the crash? It might be related to rarely used options that may not get tested as much.

Also, how does the crash manifest? When you say ‘my computer crashes’, I have visions of the dreaded Blue Screen of Death, requiring a full reset of your system. If that were the case, I would suggest the problem is with your computer - there is nothing that dwi2tensor can do that would cause the system to crash outright.

If it’s just a segmentation fault or some similar problem, it would be helpful if you could copy/paste the full command and its output, it may contain clues as to where the problem might lie.

Finally, are the data stored on an external hard drive? This can sometimes cause issues, although from what you’ve said so far, it doesn’t sound like this would be issue. But best to check, just in case.

3

Thank you helping me on what information to provide:

Here are the contents of the release/config file

!/usr/bin/python

autogenerated by MRtrix configure script

configure output:

MRtrix build type requested: release

Detecting OS: windows
Checking for C++11 compliant compiler [g++]: 6.3.0 - tested ok
Detecting pointer size: 64 bit
Detecting byte order: little-endian
Checking for variable-length array support: yes
Checking for non-POD variable-length array support: yes
Checking for zlib compression library: 1.2.11
Checking for Eigen 3 library: 3.3.0
Checking Eigen 3 memory alignment requirements: OK
Checking shared library generation: yes
Checking for Qt moc: moc (version 5.6.2)
Checking for Qt qmake: qmake (version 5.6.2)
Checking for Qt rcc: rcc (version 5.6.2)
Checking for Qt: 5.6.2

PATH = r’C:\msys64\mingw64\bin;C:\msys64\home\Orjan\mrtrix3\release\bin;C:\msys64\home\Orjan\mrtrix3\release\lib;C:\msys64\home\Orjan\mrtrix3\scripts;C:\msys64\mingw64\bin;C:\msys64\usr\local\bin;C:\msys64\usr\bin;C:\msys64\usr\bin;C:\Windows\System32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\msys64\usr\bin\site_perl;C:\msys64\usr\bin\vendor_perl;C:\msys64\usr\bin\core_perl’
obj_suffix = '.o’
exe_suffix = '.exe’
lib_prefix = ''
lib_suffix = '.dll’
cpp = [ ‘g++’, ‘-c’, ‘CFLAGS’, ‘SRC’, ‘-o’, ‘OBJECT’ ]
cpp_flags = [ ‘-std=c++11’, ‘-mno-avx’, ‘-pthread’, ‘-DMRTRIX_WINDOWS’, ‘-mms-bitfields’, ‘-Wa,-mbig-obj’, ‘-march=native’, ‘-DMRTRIX_WORD64’, ‘-isystem’, ‘C:/msys64/mingw64/include/eigen3’, ‘-Wall’, ‘-O2’, ‘-DNDEBUG’ ]
ld = [ ‘g++’, ‘OBJECTS’, ‘LDFLAGS’, ‘-o’, ‘EXECUTABLE’ ]
ld_flags = [ ‘-pthread’, ‘-Wl,–allow-multiple-definition’, ‘-lz’ ]
runpath = ''
ld_enabled = True
ld_lib = [ ‘g++’, ‘OBJECTS’, ‘LDLIB_FLAGS’, ‘-o’, ‘LIB’ ]
ld_lib_flags = [ ‘-pthread’, ‘-shared’, ‘-pthread’, ‘-Wl,–allow-multiple-definition’, ‘-lz’ ]
eigen_cflags = [ ‘-isystem’, ‘C:/msys64/mingw64/include/eigen3’ ]
moc = 'moc’
rcc = 'rcc’
qt_cflags = [ ‘-pipe’, ‘-fno-keep-inline-dllexport’, ‘-march=nocona’, ‘-mtune=core2’, ‘-Wa,-mbig-obj’, ‘-O2’, ‘-std=gnu++0x’, ‘-frtti’, ‘-Wall’, ‘-Wextra’, ‘-fexceptions’, ‘-mthreads’, ‘-DUNICODE’, ‘-DQT_NO_DEBUG’, ‘-DQT_OPENGL_LIB’, ‘-DQT_SVG_LIB’, ‘-DQT_WIDGETS_LIB’, ‘-DQT_GUI_LIB’, ‘-DQT_CORE_LIB’, ‘-DQT_NEEDS_QMAIN’, ‘-isystem’, ‘C:/msys64/mingw64/include/QtOpenGL’, ‘-isystem’, ‘C:/msys64/mingw64/include/QtSvg’, ‘-isystem’, ‘C:/msys64/mingw64/include/QtWidgets’, ‘-isystem’, ‘C:/msys64/mingw64/include/QtGui’, ‘-isystem’, ‘C:/msys64/mingw64/include/QtCore’, ‘-isystem’, ‘release’, ‘-isystem’, ‘C:/msys64/mingw64/share/qt5/mkspecs/win32-g++’ ]
qt_ldflags = [ ‘-Wl,-s’, ‘-Wl,-subsystem,windows’, ‘-mthreads’, ‘-lglu32’, ‘-lopengl32’, ‘-lgdi32’, ‘-luser32’, ‘-lmingw32’, ‘-lqtmain’, ‘-lQt5OpenGL’, ‘-lQt5Svg’, ‘-lQt5Widgets’, ‘-lQt5Gui’, ‘-lQt5Core’ ]
nogui = False

Since I tried reinstalling everything again yesterday just in case (before posting) the version should be the most recent one:

dwi2tensor.exe -version
== dwi2tensor 0.3.15-388-g261e44f8 ==
64 bit release version, built Jan 30 2017, using Eigen 3.3.0

The call is simply:
dwi2tensor -force -mask brainmask.mif -fslgrad bvect bvalt dwi.mif dti.mif

The computer actually shuts down and restarts. It runs around 30% and then smack. This does indeed suggest that there is something not working right on the computer, but it only happens in interaction with the multithreading option in MRtrix3. However, I haven’t tested it thoroughly using other software. ANTs registrations e.g. seem to be running fine. The data is on an internal hard drive. I will try putting the data on an SSD later today. I will also try this Microsoft hotfix.

4

Hi @Orjan,

There’s a good chance the issue is due to the use of Eigen version 3.3. It comes with a number of optimisations that require a greater degree of conformance on the part of developers, that if not in place can cause all sorts of issues (frankly I’m surprised it gets to 30% before crashing); hence why we instruct against it in our documentation (though admittedly it looks like it wasn’t updated once Eigen 3.3 came out of beta).

The imminent 0.3.16 version release should make MRtrix3 compatible with Eigen 3.3; for now, you can try installing Eigen 3.2, configuring MRtrix3 to use it (use the EIGEN_CFLAGS environment variable to point to the correct version of Eigen to use) and re-building. Let us know whether or not that sorts out the issue for you.

Rob

5

I doubt that. If this is a recent install, it won’t even be using the AVX instructions that have been causing us so much trouble (in fact, you can see the -mnoavx option in the cpp_flags). But more to the point, we’re talking about an overt kernel-level crash here. There’s nothing that any of our commands do that should cause Windows to BSOD like this.

This must be a hardware issue (or possibly drivers). There’s lot of candidate culprits here, but the fact that it runs without multi-threading probably means that it’s unlikely to be faulty RAM (although I’d still strongly recommend testing this thoroughly, e.g. this way). It could be a faulty CPU, where one of the cores is a bit unreliable under load. Another likely cause is simply that the CPU overheats when running too many threads (you’re not overclocking your system, are you…?), which either introduces bit errors causing a kernel-level panic, or exceeds the temperature safety threshold set in the BIOS, triggering an immediate shutdown. On the software driver side, there is potential for a buggy BIOS or chipset drivers having issues handling some of the newer instructions that Eigen 3.3 might generate, so it’s definitely a good idea to update all of your drivers and your BIOS to the latest version available. For instance, Intel release microcode to fix bugs in their CPUs, so updating their drivers might sort this out.

But either way, I don’t see that this can be in any way related to MRtrix3 as such, other than the fact that it might push your CPU harder than the previous release (either in terms of thermal load, or by using newer instructions).

6

Just a few more suggestions:

I wouldn’t bother. I’d be suspicious if you were running on an external drive, but if it’s already internal, I don’t see how this can be related. Besides, external drives can cause the application to crash or abort, but we’ve never encountered an overt system crash like this.

Again, I probably wouldn’t bother. It’s supposed to fix issues with crashes at the application level, no mention of a full system crash like you’re reporting here.

7

The eigen version does not matter, however switching to an SSD does appear to be a successful workaround in my case! I’ve subjected both the CPU and HDD to diagnostic tests (Intel and Seagate diagnostic tools, respectively) and everything checks out. All drivers and the BIOS are up-to-date. I guess there are read-write problems when multi-threading? Anyway, this sounds like it could be very specific to my system and maybe it’s best to just leave it for now, at least until someone else experiences the same issue. I’m happy working from the SSD.
Cheers!
Örjan

9

Is it still true that using SSD vs. standard HD doesn’t matter?

If I’m buying a new computer with MRtrix in mind, is there any advantage in ordering an SSD drive? If it is only a question of speeding up things, how big is the improvement for FBA? for tractogram construction? other applications?

Best,
Oren

10

Personally I would never again purchase or build another system, laptop or desktop, without an SSD. Not for MRtrix3, but for everything.

Don’t think it would have a huge effect on FBA or tractogram construction. Maybe if you were to run population_template with the scratch directory on an SSD it’d be faster, but other steps I don’t think it would matter; e.g. fixelcfestats is RAM dependent, & much of the slow execution is cache misses, which an SSD won’t solve. Tracking with 32 threads still CPU bottlenecks rather than overloading an HDD output.