I’ve been running the population_template command on a set of 46 FODs and corresponding masks for a week now. Linear registration was completed successfully, but it suddenly failed during stage 5 of 16 of non-linear registration on the mrregister command. This is the error message I got:
If anyone could help shed some light on this, I would really appreciate it! I may try to rerun the command with only 20 FODs to try to speed things up, but I want to make sure this doesn’t happen again.
The lack of output of the failed command is a bit surprising and unfortunately makes debugging this guesswork. @rsmith@jdtournier Any idea what scenarios cause no error message being shown?
You should be able to continue the script with the -continue option and a unique file identifier of the last successfully processed file (for instance warps_<X>/<filename>.mif if the last successful command was a call of mrregister). You can find the command log in the scratch directory. I’d make a backup of the temporary scratch directory before rerunning with -continue and first try to find out what went wrong.
A week processing time for 46 images does sound fairly long. Also, 20 subjects will be about twice as fast but the slowest and most memory intense steps of population_template are the last nonlinear registration stages that you haven’t even reached yet. Could your computer be running out of RAM? If so, then mrregister might have been killed by the operating system if it also ran out of swap space. You could test that by manually repeating the failed command from the scratch directory while monitoring your memory usage (via htop for instance or via egrep 'Mem|Cache|Swap' /proc/meminfo). If you run out of RAM, check the size of your transformed input image and template (mrinfo).
Thanks for the reply! I tried rerunning the mrregister command and I believe it was a RAM issue. Would cutting down the number of subjects to 20 resolve this problem?
Unlikely, registration is performed iteratively on image and template pairs so the runtime depends roughly linearly on the number of subjects, memory requirements are about constant. On any recent computer with “normal” image resolution and extent you shouldn’t run out of RAM. If this is run on a HPC node, you might need to allocate more RAM.
How large (voxels) are your input images (mrinfo) and did you request a custom template resolution (-voxel_size)? Did you upsample your input data? Make sure to use masks for all input images as this speeds up registration and helps in reducing the template size. The template size is determined after initial alignment and cropped to the dilated template mask.
@rsmith@jdtournier Any idea what scenarios cause no error message being shown?
In case it’s relevant, this must be a version of MRtrix3 prior to 3.0.0. The output error string is now changed to be more explicit in cases where the failed command does not produce any information on stdout or stderr, precisely because in cases like this it makes the script’s output message slightly confusing.
It’s mostly there for invoked external commands though. If mrregister terminated without producing any terminal output, then not even the signal handler completed; which would suggest that the operating system SIGKILLed it.
