Advanced debugging of MRtrix3 binaries

rsmith · May 23, 2019, 6:18pm

On rare occasions, a user may encounter a critical error (e.g. “Segmentation fault”) within an MRtrix3 command that does not give sufficient information to identify the cause of the problem, and that the developers are unable to reproduce. In these cases, we will often ask to be provided with example data that can consistently reproduce the problem in order to localise the issue. An alternative is for the user
to perform an initial debugging experiment, and provide us with the resulting information. The instructions for doing so are below.

If required, install “gdb”: the GNU Debugging Tool (specific instructions for this installation will depend on your particular operating system). If using macOS, the equivalent debugging tool is “lldb”, which comes with the installation of Xcode.
Make sure you are using the most up-to-date MRtrix3 code!
(git pull)
Configure and compile MRtrix3 in debug mode:
```
$ ./build select debug
$ ./configure -debug -assert
$ ./build bin/command
```
(replace “command” with the name of the command you wish to compile).

Note that this process will move your existing MRtrix3 compilation into a temporary directory. This means that your compiled binaries will no longer be in your PATH; but it also means that later they can be quickly restored without re-compiling all of MRtrix3. In addition, we only compile the command that we need to test (replace “command” with the name of the command you intend to test).
Execute the problematic command within gdb:
```
$ gdb --args bin/command (arguments) (-options) -debug
```
or lldb on macOS:
```
$ lldb -- bin/command (arguments) (-options) -debug
```
(replace “command” with the name of the command you wish to run)

The preceding “gdb --args” or “lldb --” at the beginning of the line is simply the easiest way to execute the command within gdb or lldb. Include all of the file paths, options etc. that you used previously when the problem occurred. It is also recommended to use the MRtrix3 -debug command-line option so that MRtrix3 produces more verbose information at the command-line.
If running on Windows, once gdb has loaded, type the following into the terminal:
```
$ (gdb) b abort
$ (gdb) b exit
```
These ‘breakpoints’ must be set explicitly in order to prevent the command from being terminated completely on an error, which would otherwise preclude debugging once an error is actually encountered.
At the gdb or lldb terminal, type “r” and hit ENTER to run the command.
If an error is encountered, gdb or lldb will print error information, and then provide a terminal with “(gdb)” or “(lldb)” shown on the left hand side. Type “bt” and hit ENTER: This stands for “backtrace”, and will print details on the internal code that was running when the problem occurred.
Copy all of the raw text, from the command you ran in instruction 3 all the way down to the bottom of the backtrace details, and post it to the location where the issue was first raised (either here on the community forum, or in the Issue tracker for the GitHub repository.

If gdb or lldb does not report any error, it is possible that a memory error is occurring, but even the debug version of the software is not performing the necessary checks to detect it. If this is the case, you can also try using Valgrind, which performs a more exhaustive check for memory faults (and correspondingly, the command will run exceptionally slowly):

$ valgrind bin/command (arguments) (-options)

(replace “command” with the name of the command you wish to run).

When you have finished debugging, restore your default MRtrix3 compilation:

$ ./build select default

Binaries compiled in debug mode run considerably slower than those compiled using the default settings (even if not running within gdb or lldb), due to the inclusion of various symbols that assist in debugging and the removal of various optimisations. Therefore it’s best to restore the default configuration of MRtrix3 for your ongoing use of the software.