I am trying to run
-continue option. It seems to be skipping the
dirstat command and failing (screenshot below) on the
len(shell_bvalues)==len(shell_asymmetries) + 1 check (line 297 of
dwifslpreproc) even though that condition would be met if the dirstat command was run. Is there an option to make sure
dirstat command runs through even when using the
A little bit of context on why I need to use the
option. I am using local condor setup to launch a batch of jobs running
portion seems to use less memory and only a single cpu while
can use multiple cpus and needs more memory. If I request cpus and memory sufficient for
upfront then I am not able to get optimal throughput. My strategy is to first request only a single CPU and small amount of memory to get as many
running at the same time as possible and let the
fail on out-of-memory error. Then re-launch
-continue scratch eddy_in.nii
option requesting more cpus and memory. I did use
option for the first launch.
Your help and advice on getting the
-continue option working properly or any other way of getting maximal throughput using
dwifslpreproc (because of the inherent differences in the throughputs of
eddy_openmp) would be greatly appreciated! I am currently not able to share the sample data since that would require me to include only skull-stripped data and organize about 8 Gb of data. But if that’s something you would need to help me, I will work on making that available. Please let me know.
Thanks in advance for your help!
-continue option is quite a hack, and is most certainly not guaranteed to work in all instances. I’ve often thought about how it could potentially be improved (e.g.), but with the relatively simplistic Python API for which the MRtrix3 scripts are written it’s much more difficult to provide such a mechanism in a robust fashion as opposed to something like the task scheduler in
This error in particular could probably be addressed by adding a keyword argument to the
run.function() functions to indicate that the command should always be executed, even if the
-continue option was utilised. However such a change is unlikely to fix all issues with the
-continue option. Storing the
stderr outputs of all executed commands in the scratch directory, and returning those whenever the
-continue option results in a command / function not being executed, would possibly address a greater number of limitations, but I still can’t guarantee a priori that it would lead to successful execution. You would probably also want a standard command-line option that results in script termination upon encountering a particular command, designed to be used in conjunction with
If you have motivation to get this working due to your processing needs and know a little Python, I can provide some suggestions on how to go about it, maybe over on GitHub. Personally I’m probably unlikely to get to it in the near future however
I understand! I am not a Python guru but am motivated to give it a shot, with your guidance. What should I do next to get started on this?
I’ve written some thoughts on GitHub if you want to have a go at making some modifications to the API to achieve what you need. If you make such modifications in
git based on a fork of the MRtrix3 repository, you may then get the opportunity to propose a Pull Request to merge the changes into a future MRtrix3 update.
This is exciting. I am new to such kind of work but will try to do my best to contribute. I will start digging soon.