I am trying to run dwifslpreproc with -continue option. It seems to be skipping the dirstat command and failing (screenshot below) on the len(shell_bvalues)==len(shell_asymmetries) + 1 check (line 297 of dwifslpreproc) even though that condition would be met if the dirstat command was run. Is there an option to make sure dirstat command runs through even when using the -continue option?
A little bit of context on why I need to use the -continue option. I am using local condor setup to launch a batch of jobs running dwifslpreproc. The topup/applytopup portion seems to use less memory and only a single cpu while eddy_openmp can use multiple cpus and needs more memory. If I request cpus and memory sufficient for eddy_opemp upfront then I am not able to get optimal throughput. My strategy is to first request only a single CPU and small amount of memory to get as many topups running at the same time as possible and let the eddy_openmp part of dwifslpreproc fail on out-of-memory error. Then re-launch dwifslpreproc with -continue scratch eddy_in.nii option requesting more cpus and memory. I did use -nocleanup option for the first launch.
Your help and advice on getting the -continue option working properly or any other way of getting maximal throughput using dwifslpreproc (because of the inherent differences in the throughputs of topup and eddy_openmp) would be greatly appreciated! I am currently not able to share the sample data since that would require me to include only skull-stripped data and organize about 8 Gb of data. But if that’s something you would need to help me, I will work on making that available. Please let me know.
Thanks in advance for your help!
Sincerely,
Nagesh
The -continue option is quite a hack, and is most certainly not guaranteed to work in all instances. I’ve often thought about how it could potentially be improved (e.g.), but with the relatively simplistic Python API for which the MRtrix3 scripts are written it’s much more difficult to provide such a mechanism in a robust fashion as opposed to something like the task scheduler in fmriprep.
This error in particular could probably be addressed by adding a keyword argument to the run.command() and run.function() functions to indicate that the command should always be executed, even if the -continue option was utilised. However such a change is unlikely to fix all issues with the -continue option. Storing the stdout & stderr outputs of all executed commands in the scratch directory, and returning those whenever the -continue option results in a command / function not being executed, would possibly address a greater number of limitations, but I still can’t guarantee a priori that it would lead to successful execution. You would probably also want a standard command-line option that results in script termination upon encountering a particular command, designed to be used in conjunction with -continue.
If you have motivation to get this working due to your processing needs and know a little Python, I can provide some suggestions on how to go about it, maybe over on GitHub. Personally I’m probably unlikely to get to it in the near future however
I’ve written some thoughts on GitHub if you want to have a go at making some modifications to the API to achieve what you need. If you make such modifications in git based on a fork of the MRtrix3 repository, you may then get the opportunity to propose a Pull Request to merge the changes into a future MRtrix3 update.
