I started yesterday a tckgen with 100M tracks, exactly as in the connectome tutorial. In 24 hours it is at 35%. How long does this step take for other people? I am not sure if my computer has troubles (it was slow for some reason before starting tckgen), or whether this is just normal time for tckgen.
Just checked my SIFT2 paper; I was getting 100M with dynamic seeding in 16 hours on a 4th-generation i7 processor.
A few factors that you can check / will influence this:
The processor age, number of cores, capability for hyper-threading, and cache sizes will all influence this speed.
Check your task manager and see if all of your CPU resources are being used. The CPU% reported should be close to (200 x number_of_cores) (You can get up to 200% usage per core due to the use of hyper-threading). If this isn’t utilizing all of your cores, you should try explicitly setting the number of threads in the MRtrix config file.
The most likely explanation for the difference in speed: The IIT HARDI atlas is 1mm isotropic resolution, as opposed to ~ 2mm for typical in vivo acquisitions. This presents a double-whammy for performance:
The algorithm has to fetch FOD data for a greater number of voxels per mm tracked.
The cache hit rate is likely to be reduced due to the increased image dimensions, so the processors spend more time idly waiting for data incoming from RAM. You may be able to improve this ever-so-slightly by using the mrcrop command to remove some of the dead space around the brain.
In the executing command, you omitted explicitly setting the step size. This will default to half a voxel, or 0.5mm; so the algorithm is taking twice as many steps per mm compared to my data.
So I’d say what you’re observing is probably the speed to be expected from tracking on this data with these parameters…
Would it be ok to kill the process, restart the computer and run the remaining 70M tracks?
Yes, this should be fine. The code responsible for writing .tck files should leave the file in a valid state at all times, so prematurely terminating the process should never result in a corrupt file. tckinfo will quote the number of tracks reported in the header (‘count’ field), and you can verify that this is in fact the true number of tracks in the file using tckinfo -count. The tckedit command can be used both to truncate the number of tracks in a file, and concatenate track data from multiple files.
I tried setting it together with angle threshold, but there is an error. I thought -act did not accept those settings. You can see the error above as “expected exactly 2 arguments (4 supplied)”.
That error occurred because you specified -step 1 angle 45 instead of -step 1 -angle 45 (note the missing ‘-’).
Without the preceding dash to indicate that ‘angle’ is an option (and, subsequently, that the ‘45’ is the value provided for the ‘angle’ option), the MRtrix command-line parser thinks that both ‘angle’ and ‘45’ are compulsory positional arguments rather than options. Therefore, because the tckgen command expects just two compulsory arguments (input image and output track file paths), but you have provided 4, it doesn’t know how to interpret what you have provided, and has no choice but to throw an error.
That’s great, Rob. It’s much faster now at step=1. Are .tck files with different step sizes “blendable” with tckedit?
Fyi, tckgen is using 92% of my two CPUs Xeon E5-2609v2 2.5Ghz, and I see the typical symptoms of full core/thread usage: computer is slow in everything else (reminds me of Windows 95).
I can’t find mrtrix.conf though (I am on MINGW in Windows). Is there a way to make MRtrix print on screen settings that is using (NumberOfThreads, etc) ?
Are .tck files with different step sizes “blendable” with tckedit?
They are; the output track file should quote its step size as ‘variable’. Note though that if you have tracks at 0.5mm and 1.0mm step sizes, and both used a 45-degree angle limit, then the minimum radius of curvature will be different between the two runs, which could influence the tracking of highly curved bundles.
I can’t find mrtrix.conf though (I am on MINGW in Windows).
Because MRtrix is compiled from source (ie. there’s no ‘installation’ per se), there won’t be a configuration file there unless you make one. With the most up-to-date code, running any command with the -debug option should prompt the location(s) in which the library is looking for such a file.
Is there a way to make MRtrix print on screen settings that is using (NumberOfThreads, etc) ?
There isn’t a mechanism for printing the settings being used, because the library doesn’t explicitly store values for all such settings. It only stores whatever it finds in any config file; and then when a particular setting value is requested, it is either loaded from that storage (if present), or the hard-coded default is used. So producing such a comprehensive list in the code would actually be slightly difficult.
The config file documentation lists the default value for each setting, which is what will be used if you do not manually change the value yourself using a config file.
I am getting out of the scope of this mailing list. But…
Thinking of this aspect, could it be a “feature” rather than a limitation. Has anyone compared things such as graph theory measures at different step sizes / angles? Could it be that combining tracks at different steps size / angle can provide a connectome that is more independent to those settings ?
Looks like a connectome from combined tractography results is better.
It’s important though to appreciate where this benefit is coming from. Methods such as LiFE and SIFT are looking for a tractogram that fits the image data; having tracks with multiple parameters simply increases the variability in the tractogram for methods such as these to work with during their optimisation. Naively concatenating tractograms with different parameters is unlikely to give much benefit.
Is it ok to run sift on a .tck file with combined different step sizes (0.5, 1,2, 4mm)?
It should be OK. The streamline-to-voxel mapping mechanism used by a few MRtrix commands (including tcksift) is almost step size independent. What I don’t know is whether tcksift will give an error; it will try to automatically calculate an apprioriate upsampling factor for the streamlines data so that the length through each voxel is calculated as an arc length rather than a straight line length, and I’m not sure what will happen here if the step size is marked as ‘variable’. Give it a go!
Just to follow up on this, I haven’t run the ensamble tractography. In fact, I had some problems to make tcksift work because it crashed at a random number of tract mapping each time.
It just stopped mapping tracks in the image, no error appeared, and the process disappeared from the list in task manager. One time it managed to finish mapping all tracks but there were no iterations starting.
I think this is related to the number of threads being used, or the fact that I have two processors on my machine, because once I specified -nthread 2 it ran normally and is still running.
When you say that it ‘crashed’, did it pause and fail to make any more progress, or did it terminate and go back to the terminal of its own accord?
If the former, this is a bug in the Windows implementation of a particular C++ class used for multi-threading that still hasn’t been fixed. I’ve reduced the load on that particular segment of code as much as possible (it’s a multi-threading race condition), but it does seem to still crop up from time to time, exclusively on Windows and particularly on high performance hardware. Reducing the number of threads will reduce the likelihood of the issue appearing, but can’t guarantee that it will never happen (unless you use -nthreads 0). I think I might need to implement a hack fix to try to stop this from cropping up…
If the latter, this is your system running out of RAM. This is made more likely by not only a large number of tracks, but also the high spatial resolution of the FOD image, similarly to HCP data. There’s some guidance here if this is the case.
If it’s managed to complete the ‘mapping tracks to image’ segment with a reduced number of threads, it’ll be #1. If it never completes that section regardless of the number of threads, it’ll be #2.
It is very likely to be #1. The terminal keeps showing the command running, the prompt doesn’t come back. But effectively the process has crashed and there is no advancement. As you predicted, -nthreads 2 solved the problem and sift is finally converging to the number I asked (10M), albeit is taking 2 days.
The most unusual thing is that there is no error or return of prompt, which makes the user think the process is still running. Usually when errors occur in a software something happens, in this case it just stops advancing with no other sign of crash.
@rsmith: sounds like this issue again, hey? Sounds like we may need to write a minimal example in straight C++11 to demonstrate that the problem has nothing to do with MRtrix, and submit a bug report to the MinGW developers. This only happens on Windows, it really is most likely to be a compiler bug…
That certainly did the trick back when I looked into it.
I suspect writing a MWE to send to MinGW might actually be a bit of work. But I suppose I’ll have to put in the time given they don’t appear to have found it yet. Grumph.
