I’m just curious as to whether there has been a change in the way the “for_each” command handles batch processing across the last couple of releases?
Rather than outputting the status of each process individually, it now updates only the percentage of jobs completed and displays errors only at the end of the task.
eg for_each [ 5%] 2/39 jobs completed sequentially…
This may be a local python environment thing, but wanted to ask anyway as the batch commands now seem to be exciting a low slower than previously.
From 3.0_RC3 to 3.0.0, for_each was completely re-written from Bash into Python (with some additional functionalities) as part of a wider Python API overhaul. Previously, post-substitution commands were executed via xargs and wrote their stdout / stderr contents directly to the terminal, which resulted in a mess if trying to find error messages when a subset of such jobs fail and/or if multiple jobs were run in parallel. The standard MRtrix3-info and -debug options change verbosity if you want to be able to see more info about commands being executed. Displaying stdout / stderr outputs of commands as they execute would require a bit more development work on that script.
While I’d expect the overhead of preparing commands for execution in the new implementation to be greater, I’d only expect this to be consequential in circumstances where each individual invocation of the command being executed is exceptionally fast. It’s possible that it may be a coincidental correlation between execution speed of whatever you’re doing and the for_each changes; but you could easily test that by simply grabbing the old version and executing it as a standalone bash script.