-continue option failing with population template

Hello,

I am currently trying to follow the fixel-based analysis steps (multi tissue pipeline) and am stuck at step 9 (FOD population template). I work with a time-limited server and the population template command runs beyond the limit so I want to use the -continue option by specifying the tmp directory once it’s interrupted.

However I can’t resume the script. I run the following

population_template MYPATH/template/fod_input -mask_dir MYPATH/template/mask_input MYPATH/template/wmfod_template.mif -voxel_size 1.25 -nthreads 16 -scratch MYPATH/template/ -nocleanup -continue MYPATH/template/population_template-tmp-FGDGRV/ MYPATH/template/population_template-tmp-FGDGRV/linear_transforms_07/LASTSUBJECT.txt"

But then it returns an error that seems to indicate it is looking for a file that has not been generated yet (the first subject from linear_transforms_08, which is empty, because 07 wasn’t completed yet). A bunch of commands are skipped beforehand but I don’t understand their purpose so I if it’s not normal you will be able to flag it:

population_template: Changing to scratch directory (MYPATH/template/population_template-tmp-FGDGRV)
population_template: Generating initial template
Skipping command: mraverageheader -fill [MYPATH/template/fod_input/0*.mif (69 items)] - | mrgrid - regrid -voxel 1.25,1.25,1.25 average_header.mif
population_template: Importing input masks to average space for template cropping… [=]
Skipping command: mrmath [mask_transformed/.mif (69 items)] max mask_initial.mif
Skipping command: mrgrid average_header.mif crop -mask mask_initial.mif average_header_cropped.mif
Skipping function: posix.remove(‘mask_initial.mif’)
Skipping function: posix.remove(‘average_header.mif’)
Skipping function: shutil.move(‘average_header_cropped.mif’, ‘average_header.mif’)
population_template: Erasing temporary mask images… [=]
Skipping function: shutil.move(‘average_header.mif’, ‘average_header_c0.mif’)
population_template: Performing initial rigid registration to template… [==]
Skipping command: mrgrid average_header_tight.mif pad -uniform 10 - | mrgrid - regrid -voxel 1.25,1.25,1.25 average_header.mif
Skipping function: posix.remove(‘average_header_tight.mif’)
population_template: Reslicing input masks to average header… [=]
Skipping command: mrmath [mask_transformed/
.mif (69 items)] max mask_translated.mif
Skipping command: mrgrid average_header.mif crop -mask mask_translated.mif average_header_cropped.mif
Skipping command: mrgrid average_header_cropped.mif pad -uniform 10 average_header.mif
Skipping function: posix.remove(‘average_header_cropped.mif’)
population_template: Reslicing masks to new padded average header… [=]
Skipping function: posix.remove(‘mask_translated.mif’)
population_template: Reslicing input images to average header… [=]
Skipping command: mrmath [input_transformed_c0/*.mif (69 items)] mean -keep_unary_axes initial_template_c0.mif
population_template: Optimising template with linear registration (stage 1 of 12; rigid)… [=========
population_template: [ERROR] Unhandled Python exception:
population_template: [ERROR] IOError: [Errno 2] No such file or directory: ‘linear_transforms_08/FIRSTSUBJECT.txt’
population_template: [ERROR] Traceback:
population_template: [ERROR] MYPATH/.conda/envs/mrtrix3env/bin/population_template:1237 (in execute())
population_template: [ERROR] transform = matrix.dot(matrix.load_transform(os.path.join(‘linear_transforms_%02i’ % level, inp.uid + ‘.txt’)), average_inv)
population_template: [ERROR] MYPATH/.conda/envs/mrtrix3env/lib/mrtrix3/matrix.py:135 (in load_transform())
population_template: [ERROR] data = load_matrix(filename, **kwargs)
population_template: [ERROR] MYPATH/.conda/envs/mrtrix3env/lib/mrtrix3/matrix.py:124 (in load_matrix())
population_template: [ERROR] data = load_numeric(filename, **kwargs)
population_template: [ERROR] MYPATH/.conda/envs/mrtrix3env/lib/mrtrix3/matrix.py:103 (in load_numeric())
population_template: [ERROR] with open(filename, ‘rb’) as infile:

According to the log.txt, the script was doing mrtransforms to output files in linear_transforms_07 when it stopped:

transformcalc linear_transforms_07/SUBJECT.txt decompose linear_transforms_07/SUBJECT.txtdecomp
posix.remove(‘linear_transforms_07/SUBJECT.txtdecomp’)
transformcalc linear_transforms_07/SUBJECT.txt linear_transforms_07/SUBJECT.txt linear_transforms_07/SUBJECT.txt [rest of my subjects] average linear_transform_average.txt -quiet
transformcalc linear_transform_average.txt invert linear_transform_average_inv.txt -quiet
mrtransform -reorient_fod yes MYPATH/template/fod_input/SUBJECT.mif -template linear_template06_c0.mif -linear linear_transforms_07/SUBJECT.txt input_transformed_c0/SUBJECT.mif -nan -datatype float32
[repeated mrstransform interrupted at one of the subjects]

But inputting the last generated mrtransform output in 07 does not make it resume the mrtransforms for the next subjects, which is what I expect.

I had a look at older related threads but couldn’t figure out what my issue was.
I tried

  • updating mrtrix
  • making sure I’m using the full path to specify scratch dir/file after the -continue option
  • inputting files generated earlier; from ‘linear_transforms_06/7’ (the file right before the last mrstransformed file, or earlier ones in 06), ‘input_transformed_c0’ directories; I also tried inputting ‘linear_template06_c0.mif, linear_transform_average.txt’, linear_transform_average_inv.txt’. (it keeps asking for that 08 file no matter what)
  • copying files from linear_transforms_07 to 08 (it then asks for the first subject in 09 and so on)

Am I using the continue option correctly? Is there another file I should have defined as last generated file?

Thanks,

Charly