Population_template -continue option not working


#1

Dear experts,

I am following the steps for the Fixel-Based Analysis and I need to generate a population template for my 49 FODs. The init_nl_template_mask.mif was created and the script created the warps_0/subject1.mif file before power maintenance resulted in power loss to my computer.

I am trying to run the population template again using the -continue option. However, I had an error that the linear_transforms_0/subject1.txtdecomp could not be found. I manually created the .txtdecomp files for linear_transforms_0 to linear_transforms_5. However, it keeps bugging stating there is no such file or directory for warps_15/subject1.mif. Could you please tell me how I should go about this. I don’t want to start from the beginning since it has already taken 1 week to get to this point.

Thanks.
Isaac.


#2

Hello Isaac,

The string you pass on as the last successful command to the -continue option in population_template has to uniquely define the step. So, for instance don’t use subject1.mif if you can use warps_0/subject1.mif. Happy to assist if you post the output of the script.

Also, make sure that warps_0/subject1.mif and input_transformed/subject1.mif are not empty images.

The bug related to missing txtdecomp files is going to be fixed soon (mrtrix version 0.3.16).

Cheers,
Max


#3

Hi Max,

This is the command I use:
population_template all_FOD -mask_dir mask /home/isaac/AFD/template/FOD_template.mif -continue /home/isaac/afd_temp/population_template-tmp-PVGNDK/ /home/isaac/afd_temp/population_template-tmp-PVGNDK/warps_0/subject1.mif

I have also verified that the input_transformed/subject1.mif is not empty.

The output of the error message is:

File “/mrtrix3_src/mrtrix3_multi_deb7/release/scripts/population_template”, line 528, in <module>

shutil.move(‘warps_%i/%s.mif’ % (level, i.prefix), ‘warps/’)
File “/usr/nir/src/pthon2.7_anaconda_deb/anaconda2/lib/python2.7/shutil.py”, line 302, in move

copy2(src, real_dst)
File “/usr/nir/src/pthon2.7_anaconda_deb/anaconda2/lib/python2.7/shutil.py”, line 130, in copy2

copyfile(src, dst)
File “/usr/nir/src/pthon2.7_anaconda_deb/anaconda2/lib/python2.7/shutil.py”, line 82, in copyfile

with open(src, ‘rb’) as fsrc:
IOError: [Errno 2] No such file or directory: ‘warps_15/subject1.mif’

Thanks.
Isaac


#4

It skips all non-linear steps because the path you provide does not match any command which I think is because the script does not use the full path but a relative path. Have you tried warps_0/subject1.mif?

If this does not work, could you please post the full output of the script?


#5

Thank you very much Max. It was indeed a relative path and it now works.

Isaac.


#6

Yes, -continue can be quite finicky to get working (especially for a beast like population_template).

It’s not actually checking for commands trying to overwrite existing files or anything like that; it’s literally just comparing the text you provide with the -continue option against the text within each command called, to decide where/when it should resume running commands. Hence why it requires a relative file path: it’s just matching text, not the location of a file. Maybe one day I’ll do better

Glad you hear you got it going again :thumbsup:


#7

hmm @rsmith & @maxpietsch,

I appear to be having a similar issue with txtdecomp.

Our IT manager decided it was a good idea to perform maintenance on the weekend, and as a result terminating the long-running population_template.

I had made it up to nl_linear11.mif, with the last file accrued warps_12/7587.mif.

My resumed command is:

/beegfs/scratch/tnc_scratch/aperry/MRtrix/02032017/scripts/population_template /beegfs/scratch/tnc_scratch/kfo_pd_connectome/PD_connectome/AFD/FOD/WM/ -mask_dir /beegfs/scratch/tnc_scratch/kfo_pd_connectome/PD_connectome/AFD/Maskinput/ /beegfs/scratch/tnc_scratch/kfo_pd_connectome/PD_connectome/AFD/fodtemplate.mif -nthreads 10 -continue /beegfs/scratch/tnc_scratch/kfo_pd_connectome/PD_connectome/tmp/population_template-tmp-Q8DAI6/ warps_12/7587.mif -nocleanup -force

Which invokes the following error:

IOError: [Errno 2] No such file or directory: ‘linear_transforms_0/7772.txtdecomp’

I’ve also confirmed that inputs_transformed/7772.mif is not empty.

I’m not sure how to upload the error output.

Cheers,

Alistair


#8

Hello Alistair,

I’ve fixed that bug in the master branch 6 days ago. If you do not or can not update MRtrix3, then replace this line

    check_linear_transformation('linear_transforms_%i/%s.txt' % (level, i.prefix), pause_on_warn=do_pause_on_warn)

with

    if os.path.isfile('linear_transforms_%i/%s.txt' % (level, i.prefix)): # not true if -continue option is used
      check_linear_transformation('linear_transforms_%i/%s.txt' % (level, i.prefix), pause_on_warn=do_pause_on_warn)

in scripts/population_template.

Make sure to keep the indentation level consistent. See here for the current version of population_template.


#9

Hey Max,

Hey @maxpietsch,

I’ve tried both ways:

  1. Edit the line 425 in the existing population_template

  2. Update MRtrix - and copy the new population_template to the existing folder

I’m still receiving the following error:

Skipping command: transformcalc linear_transforms_0/7772.txt decompose linear_transforms_0/7772.txtdecomp
Traceback (most recent call last):
File “/beegfs/scratch/tnc_scratch/aperry/MRtrix/02032017/scripts/population_template”, line 426, in
check_linear_transformation(‘linear_transforms_%i/%s.txt’ % (level, i.prefix), pause_on_warn=do_pause_on_warn)
File “/beegfs/scratch/tnc_scratch/aperry/MRtrix/02032017/scripts/population_template”, line 72, in check_linear_transformation
data = load_key_value(transformation + ‘decomp’)
File “/beegfs/scratch/tnc_scratch/aperry/MRtrix/02032017/scripts/population_template”, line 59, in load_key_value
with open (file, “r”) as f:
IOError: [Errno 2] No such file or directory: ‘linear_transforms_0/7772.txtdecomp’

Updating the package (and exporting it to your PATH) is not a suitable option as the -continue option is looking for the version in which the original population_template was performed.

Best,

Alistair


#10

My bad, fix is on its way. Sorry for the trouble and thanks for reporting!

In the meantime, you could just comment out lines 425 and 426 as they are optional checks or wait for the pull request to finish.

Cheers,
Max

Edit: is merged to master


#11

Great @maxpietsch,

All fixed (fingers crossed).

Cheers,

Alistair


#12

Dear all,

I was wondering if, when using the option -continue, we can add new brains. In other words, is it possible to create the fod template with x brains and then add y brains and use -continue to add them to the fod template?

Thank you,
Inês


#13

I was wondering if, when using the option -continue, we can add new brains.

No. The template building process relies on loading the linear and non-linear warps estimated from previous iterations and updating them based on increasingly sharper image features, rather than re-estimating transforms from scratch at each iteration. If new subjects were added halfway through, the warps from previous iterations would not be present for those subjects, and the script would fall over.

If the -continue option were sufficiently intelligent it could theoretically figure this out and run lower-resolution registration steps for those subjects; but this:

  • Would result in a residual bias in the per-iteration population templates

  • Would require far more work than would be justified (the -continue option is fragile enough as it is…)

Rob


#14

@Ines_Rodrigues If the images you used before are representative of your population then I’d suggest to use the final template to register all images (also the ones you already used) to it and create a new template by applying mrmath transformed_images_here mean new_template. You might want to reuse the template mask.


#15

Dear MRtrix experts,

My PC crashed in the very last step of population_template:

population_template: [ 47%] Optimising template with non-linear registration (stage 16 of 16)...

and I’m trying to finish the execution it with the -continue option. I had a look at the temporary population folder and the last file succesfully created is warps_15/sub37.mif. So I try to finish with the following command:

population_template $Template_dir/fod_input -mask_dir $Template_dir/mask_input $Template_dir/wmfod_template.mif -continue /Users/adminuser/population_template-tmp-VLU0WJ/ warps_15/sub37.mif -nocleanup

However, I get the following error.txt:

posix.remove(‘warps_14/sub37.mif’)
OSError: [Errno 2] No such file or directory: ‘warps_14/sub37.mif’

Am I missing something? Obviously, warps_14/sub37.mif does not exist, because it has been warped and thus moved to the folder warps_15/. In folder warps_14/ I have sub38.mif and the following ones…the ones left to do.

I could use the last non-linear transformation computed (since I was almost there), but I’d rather learn what I’m doing wrong for the future templates that I will have to conduct.

Just in case, I updated MRtrix 5 days ago.

Thank you!


#16

Hi Ane,

That’s an unfortunate place to crash.

warps_14/sub37.mif does not exist, because it has been warped and thus moved to the folder warps_15

Close, warps_14/sub37.mif was not moved was used to initialise the registration leading to the new transformation in warps_15.

The continue option skips commands until it finds a string in the command that matches your string. The first command after that command is not skipped. This is sufficient if the script creates unique files (but does not overwrite or remove them). One way of getting around that limitation is to use a larger snipped that is unique to the last successful operation such as -option output_file.ext. Unfortunately, it will not work in your case because the registration command that creates warps_15/sub37.mif
mrregister sub37.mif nl_template14.mif ... -nl_warps_full warps_15/sub37.mif
is followed by
posix.remove(warps_14/sub37.mif)
which contains nothing but the warp filename.

Since you skip creation (registration) of warp_14/sub37.mif, you can simply copy warps_14/sub37.mif to warps_13/sub37.mif in the temporary directory so that this file can be removed successfully.

I’ll change population_template to avoid this problem in the future.

Also, you might want to check that the last created file is complete (every volume non-zero). This could have happened if your computer crashed while writing that file. In that case, you’d need to recreate that warp which requires warp_13, which in turn requires to recreate warp_12, which in turn … leaves you with a continue just after the creation of the last linearly aligned template.

Best,
Max

Edit: pull request is here


#17

Did you mean that i can skip the creation of warp_14/sub37.mif , so I can copy warps_15/sub37.mif to warps_14/sub37.mif? I guess so, because I can’t copy warps_14/sub37.mif anywhere since it doesnt exist…

Thanks for the detailed response!


#18

Yes, I meant copy warps_15/sub37.mif to warps_14/sub37.mif so that it can be deleted.