Hi Rachel,
I’ve not observed this kind of behaviour, and there’s no particular reason why the command should be running that much slower than the previous code. I’d have expected the execution time to take less than 12 hours given the total amount of processing should be less than your prior 12 separate executions, each of which individually required building the fixel-fixel connectivity matrix. The internals of the GLM have changed quite a lot, but I didn’t observe any major slowdowns in my own testing.
I can only generate a couple of hypotheses that are consistent with the information provided (there are new capabilities that will slow down execution, but they’re not relevant given their absence from your terminal output):
-
Your number of inputs is much larger than anything I’ve tested on. The change in empirical null distribution generation from Manly to Freedman-Lane will involve an additional 1208x1208 matrix multiplication compared to the old code, which won’t be super cheap.
-
The old code handled generation of t-values in batches, whereas the new code does the whole matrix multiplication for all fixels in one go. It’s possible that with the data I did testing on that was fine, but with your very large number of inputs the matrix data are becoming too large, leading to cache miss slowing down execution. I can probably re-introduce some manual buffering to the execution which might help (and if I do I now know who to ask to test it
). But I’d have hoped that Eigen would have made the appropriate decisions here… -
The fixel-fixel connectivity matrix should either be memory-mapped, or explicitly loaded into RAM if this is not possible. There’s some chance that in your situation these data are not explicitly loaded into RAM at the commencement of execution, but access to the data are also slow: this would lead to the CFE portion taking a long time to execute due to delays in acquiring the fixel connectivity information for each fixel to be enhanced.
Given the length of the permutation progress bar I’m hoping that execution has completed in the time since you made the post. But thanks for flagging, and I’ll have to think for a little while about where effort needs to be invested here.
Cheers
Rob