I recently attempted to generate a population template with 40 subjects (20 controls and 20 patients). The script ran without error for over 5 days on my machine and then finally errored out this morning and I have no idea why.
This is a possibility. I may be running out of memory or ram. I do not see a directory named warps_13, however, there is a warps_12. My temp folder appears to have space in it though it is quite full:
It’s very likely that you have run out of space on /tmp. On most modern systems /tmp is not stored on disk, but a separate location in RAM (see tmpfs). For example, here is the output of df -h on my system:
As you can see tmpfs is quite small. While having /tmp in RAM is nice and fast for things like piping images between MRtrix commands, it’s probably not the best place to store large working directories in scripts. We may end up changing the default working directory for scripts to be inside the local working directory in the future, however for now try re-running population_template and set the temp working directory (using -tempdir) to somewhere on your file system where there is lots of space.
Do you have an entry that is Mounted on /tmp in the output of df -h? If not then I think you should have enough space in / for a template of 40 subjects unless you use files that are a lot higher resolved than HCP files.
This should not happen, as all warp_xx diretories should be generated at the start of the script. What arguments did you use? You can find them in command.txt in your temporary directory.
Also, you can save yourself some time and continue running the script at the point where the error occurred without rerunning everything from the start by using the -continue option. For that, you can move the temporary directory to a location of your choosing and give that as first argument to the -continue option. The second argument must be a unique string that identifies the last successfully run command.
Previously, as we discussed, when I tried to create a population template I was running out of memory. Subsequently, I tried a smaller sample (10 patients and 10 controls for a 20 subject population template). I received the following error which I am assuming was simply because there was not a .mif at the end of the command? I tried using the -continue option to continue the analysis that took approximately 3-4 days and could not get it running. Could someone please confirm that this error is in fact only because the command did not have an ending? Also, if this is the issue it may be nice to have an initial catch for this so it’s not running for days and failing at the last step. Thank you all for the help.
You are right, we do not check whether the output is a valid image file, only if it exists. @rsmith Shall I add an optional argument to check for image file extensions to checkOutputFile?
Why did -continue not work for you?
I believe that you can change the output arguments even if you use -continue. Alternatively, go to the temporary directory and manually copy nl_template.mif and if needed the directories warps and input_transformed to the destination.
@maxpietsch: checkOutputFile() is I think already being used to check for the existence of output directories, so a direct file extension check there is not appropriate (but then again, neither is the function name any more…). Adding an extension check for just the output of population_template would probably be a good idea given that script takes a mixture of file & directory paths, so it’s likely a more common mistake. It would be an incomplete solution though; GitHub issue here could theoretically be exploited to perform more thorough checks of output files in scripts, can discuss technicals there if you want.
@jbishop: Yes, -continue can be a bit finicky. Thankfully, since in your case it’s literally just grabbing the output files from the temporary directory that has failed, you can do that manually without worrying about getting -continue to work: just go into the temporary directory and make copies of whatever you want to keep.
