Error during tckgen

1

Hi all! When running

tckgen -act OCD8_1day/5tt_coreg.mif -seed_gmwmi OCD8_1day/gmwmSeed_coreg.mif -angle 22.5 -maxlen 250 -minlen 10 -power 1.0 OCD8_1day/wmfod.mif -select 20000000 -cutoff 0.10 OCD8_1day/tracks_20_million.tck
I'm getting the following error:
tckgen: [ 1%] 136821871 seeds, 24859294 streamlines,  370453 selected
tckgen: [ 3%] 273576931 seeds, 49712398 streamlines,  741980 selected
tckgen: [100%] 444657903 seeds, 80810827 streamlines, 1206175 selected
tckgen: [ERROR] error opening output file "OCD8_Pre/lowMemHighCPU.tck": Cannot send after transport endpoint shutdown

Wondering if anyone has experienced this before and how to troubleshoot it? Not sure if it makes a difference, but this is on a HPC environment using 8 CPU’s…
Thanks in advance!

2

I’ve not come across this particular error before, but a quick Google search suggests this is probably a glitch with the lustre filesystem. You’d need to check with your system administrator on that one – it’s unlikely to be related to MRtrix3 as such…

3

Thanks a lot for responding about that! I will look into that error on our system. The other error, which I think is more likely on the MRTRIX end (or at least my use of it), is:

tckgen: [ 0%] 8473881 seeds, 1541256 streamlines,  28601 selected
tckgen: [100%] 9595002 seeds, 1745954 streamlines,  32305 selected
terminate called after throwing an instance of 'MR::Exception'

-is this a known issue?

4

OK, that one is more likely to be for us… Can you tell us:

  • what your exact command line invocation was?
  • what version you’re running (tckgen -version)?
  • what the output is when running with the -debug option?
5

Thank you for getting back to me! Sorry in advance this is going to look like a massive post, but want to give you all the info…!
My command was:

tckgen -act OCD8_1day/5tt_coreg.mif -seed_gmwmi OCD8_1day/gmwmSeed_coreg.mif -angle 22.5 -maxlen 250 -minlen 10 -power 1.0 OCD8_1day/wmfod.mif -select 20000000 -cutoff 0.10 OCD8_1day/tracks_20_million.tck

tckgen -version 
64 bit release version, built Nov  9 2018, using Eigen 3.3.5

Here’s the full output from -debug:

tckgen: [DEBUG] No config file found at "/etc/mrtrix.conf"
tckgen: [DEBUG] No config file found at "/home/benny89/.mrtrix.conf"
tckgen: [INFO] opening image "OCD8_1day/5tt_coreg.mif"...
tckgen: [DEBUG] reading key/value file "OCD8_1day/5tt_coreg.mif"...
tckgen: [DEBUG] sanitising image information...
tckgen: [INFO] image "OCD8_1day/5tt_coreg.mif" opened with dimensions 146x187x111x5, voxel spacing 0.9375x0.9375x1.2000999999999999xnan, datatype Float32LE
tckgen: [DEBUG] memory-mapping file "OCD8_1day/5tt_coreg.mif"...
tckgen: [DEBUG] file "OCD8_1day/5tt_coreg.mif" mapped at 0x2ad74e6b7000, size 60610440 (read-only)
tckgen: [DEBUG] image "OCD8_1day/5tt_coreg.mif" loaded
tckgen: [DEBUG] image "OCD8_1day/5tt_coreg.mif" initialised with strides = [ 5 730 136510 1 ], start = 0, using direct IO
tckgen: [INFO] opening image "OCD8_1day/gmwmSeed_coreg.mif"...
tckgen: [DEBUG] reading key/value file "OCD8_1day/gmwmSeed_coreg.mif"...
tckgen: [DEBUG] sanitising image information...
tckgen: [INFO] image "OCD8_1day/gmwmSeed_coreg.mif" opened with dimensions 146x187x111, voxel spacing 0.9375x0.9375x1.2000999999999999, datatype Float32LE
tckgen: [DEBUG] memory-mapping file "OCD8_1day/gmwmSeed_coreg.mif"...
tckgen: [DEBUG] file "OCD8_1day/gmwmSeed_coreg.mif" mapped at 0x2ad752085000, size 12122088 (read-only)
tckgen: [DEBUG] image "OCD8_1day/gmwmSeed_coreg.mif" loaded
tckgen: [DEBUG] image "OCD8_1day/gmwmSeed_coreg.mif" initialised with strides = [ 1 146 27302 ], start = 0, using direct IO
tckgen: [DEBUG] sanitising image information...
tckgen: [DEBUG] allocating scratch buffer for image "scratch image"...
tckgen: [DEBUG] image "scratch image" loaded
tckgen: [DEBUG] image "scratch image" initialised with strides = [ 1 137 25482 ], start = 0, using direct IO
tckgen: [DEBUG] unmapping file "OCD8_1day/gmwmSeed_coreg.mif"
tckgen: [DEBUG] image "OCD8_1day/gmwmSeed_coreg.mif" unloaded
tckgen: [INFO] opening image "OCD8_1day/wmfod.mif"...
tckgen: [DEBUG] reading key/value file "OCD8_1day/wmfod.mif"...
tckgen: [DEBUG] sanitising image information...
tckgen: [INFO] image "OCD8_1day/wmfod.mif" opened with dimensions 185x185x111x45, voxel spacing 1.3x1.3x1.3xnan, datatype Float32LE
tckgen: [DEBUG] memory-mapping file "OCD8_1day/wmfod.mif"...
tckgen: [DEBUG] file "OCD8_1day/wmfod.mif" mapped at 0x2ad75367c000, size 683815500 (read-only)
tckgen: [DEBUG] image "OCD8_1day/wmfod.mif" loaded
tckgen: [DEBUG] image "OCD8_1day/wmfod.mif" initialised with strides = [ -45 -8325 1540125 1 ], start = 1540080, using direct IO
tckgen: [INFO] opening image "OCD8_1day/5tt_coreg.mif"...
tckgen: [DEBUG] reading key/value file "OCD8_1day/5tt_coreg.mif"...
tckgen: [DEBUG] sanitising image information...
tckgen: [INFO] image "OCD8_1day/5tt_coreg.mif" opened with dimensions 146x187x111x5, voxel spacing 0.9375x0.9375x1.2000999999999999xnan, datatype Float32LE
tckgen: [DEBUG] memory-mapping file "OCD8_1day/5tt_coreg.mif"...
tckgen: [DEBUG] file "OCD8_1day/5tt_coreg.mif" mapped at 0x2ad77c2a1000, size 60610440 (read-only)
tckgen: [DEBUG] image "OCD8_1day/5tt_coreg.mif" loaded
tckgen: [DEBUG] image "OCD8_1day/5tt_coreg.mif" initialised with strides = [ 5 730 136510 1 ], start = 0, using direct IO
tckgen: [INFO] step size = 0.649999976 mm
tckgen: [INFO] maximum deviation angle = 22.5 deg
tckgen: [INFO] minimum radius of curvature = 2.5999998322809814 mm
tckgen: [INFO] iFOD2 internal step size = 0.216666654 mm
tckgen: [DEBUG] creating empty file "OCD8_1day/tracks_20_million.tck"
tckgen: [INFO] rejection sampling will use 7 directions with a ratio of 1.69752324 (predicted number of samples per step = 8.35358715)
tckgen: [DEBUG] initialising threads...
tckgen: [DEBUG] launching 32 threads "source"...
tckgen: [DEBUG] launching thread "sink"...
tckgen: [DEBUG] waiting for completion of threads "source"...
tckgen: [  0%]        1 seeds,        1 streamlines,        0 selected
tckgen: [  0%]      481 seeds,       89 streamlines,        1 selected
tckgen: [  0%]     1191 seeds,      217 streamlines,        2 selected
tckgen: [  0%]     2941 seeds,      555 streamlines,        9 selected
tckgen: [  0%]     8321 seeds,     1488 streamlines,       31 selected
tckgen: [  0%]    22931 seeds,     4188 streamlines,       85 selected
tckgen: [  0%]    54151 seeds,     9818 streamlines,      181 selected
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00750347646, max_val = 0.00743552065)
tckgen: [  0%]   118251 seeds,    21441 streamlines,      402 selected
tckgen: [  0%]   253291 seeds,    46056 streamlines,      856 selected
tckgen: [  0%]   509091 seeds,    92398 streamlines,     1689 selected
tckgen: [  0%]  1019571 seeds,   185419 streamlines,     3409 selected
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00684713898, max_val = 0.00610072073)
tckgen: [  0%]  2023431 seeds,   367572 streamlines,     6731 selected
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00293921493, max_val = 0.00277389865)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00339965266, max_val = 0.00260430505)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00289558736, max_val = 0.00257624057)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00437996676, max_val = 0.00392492907)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00859572925, max_val = 0.00850170013)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00285123917, max_val = 0.00285098422)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00324572739, max_val = 0.00280245068)
tckgen: [  0%]  4047801 seeds,   735443 streamlines,    13479 selected
tckgen: [DEBUG] max_val exceeded!!! (val = 0.0107054375, max_val = 0.0100912051)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00279993517, max_val = 0.00256919512)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00628530746, max_val = 0.00602932926)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00555938715, max_val = 0.00529832626)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00587909995, max_val = 0.00532327779)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00298301643, max_val = 0.002811688)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.0149511341, max_val = 0.012837532)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00425111828, max_val = 0.00305314432)
tckgen: [  0%]  8120701 seeds,  1477067 streamlines,    27262 selected
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00329568796, max_val = 0.0031709536)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00552264787, max_val = 0.00491045136)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00534383114, max_val = 0.00523250783)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00241314596, max_val = 0.00236748322)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00495247636, max_val = 0.00485883327)
tckgen: [DEBUG] max_val exceeded!!! (val = 0.00722431671, max_val = 0.00503202248)
tckgen: [DEBUG] no readers left on queue "source->sink"
tckgen: [DEBUG] no writers left on queue "source->sink"
tckgen: [DEBUG] threads "source" completed OK
tckgen: [DEBUG] waiting for completion of thread "sink"...
tckgen: [100%]  9669871 seeds,  1759115 streamlines,    32387 selected
terminate called after throwing an instance of 'MR::Exception'
/var/spool/slurmd/job26785997/slurm_script: line 6:  9242 Aborted                 (core dumped) tckgen -act OCD8_1day/5tt_coreg.mif -seed_gmwmi OCD8_1day/gmwmSeed_coreg.mif -angle 22.5 -maxlen 250 -minlen 10 -power 1.0 OCD8_1day/wmfod.mif -select 20000000 -cutoff 0.10 OCD8_1day/tracks_20_million.tck -debug
6

Are you sure that’s the full output? I was hoping to see something like this:

== tckgen 3.0_RC3_latest-70-gc11717d7 ==
64 bit release version with asserts, built Jan 11 2020, using Eigen 3.3.7
Author(s): J-Donald Tournier (jdtournier@gmail.com) and Robert E. Smith (robert.smith@florey.edu.au)
Copyright (c) 2008-2018 the MRtrix3 contributors.

This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, you can obtain one at http://mozilla.org/MPL/2.0/

MRtrix3 is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty
of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

For more details, see http://www.mrtrix.org/

That first line is all I’m after…

In any case, I’m not sure we’re any wiser with the debug output… The issue is how to handle errors in a multi-threaded application (see e.g. this post for context if you’re interested in the technicalities). I was hoping the changes we added at the time would at least have been sufficient to see the actual error message though…

But if your issue is anything like this one, then the chances are it’s likely to be related to errors writing out the output files (out of storage space, maybe?).

7

You’re right, there was one more line:

[benny89@gra-login3 benny89]$ tckgen -version
== tckgen unknown ==
64 bit release version, built Nov 9 2018, using Eigen 3.3.5
Author(s): J-Donald Tournier (jdtournier@gmail.com) and Robert E. Smith (robert.smith@florey.edu.au)
Copyright (c) 2008-2018 the MRtrix3 contributors.
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, you can obtain one at http://mozilla.org/MPL/2.0/
MRtrix3 is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty
of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
For more details, see http://www.mrtrix.org/

Looking at the thread you direct me to, perhaps I should look into a static version. I’m also exploring the issue of space as well, it looks like there’s plenty, but maybe there’s something I’m missing!

8

Not as informative as I was hoping…

Unfortunately this is what happens when installing from a direct download rather than a git clone (that’ll be fixed with the next release). Nothing wrong with that, just makes it harder to track the problem.

I doubt that’ll make any difference. I wouldn’t bother personally (static builds come with headaches all of their own).

Unfortunately, all we can see is that an exception was thrown. So we know the error is one that our code generated, but we can’t see what the message was. One way to investigate if you really want to get to the bottom of it is to run with -nthread 0 – it’ll be a lot slower, but if the error occurs again, you should see the actual error message…

9

I tried running with -nthread 0 but for some reason i keep hitting a time limit (despite me setting a time limit of 48 hrs), something to do with the HPC setup I’m using, seems to kick in a 1 hr time limit when I use the -nthread 0 option

10

Odd – and annoying… But to be honest, I’d be surprised if the problem wasn’t somehow related (or even the same as) your original one (Cannot send after transport endpoint shutdown). Maybe in some cases you get to see the error message, but not in others – it might depend on how the different threads interact somehow. For now, I would probably assume it’s the same problem and focus on fixing that.