Population template error at nonlinear registration stage

1

Hello All,

I am trying use 36 subjects to create the template but I got errors. Then I tried to use a series subsets to test this population_template command and found out a subset of n <= 23 could go through the processing and any subset of n > 23 failed. Here is the example of the main error message while I ran n=24 subjects (detailed output of the command to see the end ).
mrregister: performing FOD registration
mrregister: [ERROR] Permission denied
mrregister: [ERROR] error opening image “warps_00/TBI3031.mif”

What bothered me most is that questioned subject is not the newly added one. I also tried to add different subject to form the n=24 subset, the questioned subject is different.

In addition, my computer system with 64G RAM, 6.3G tmpfs, and 64T available disk space.

Can anybody help me? I’ve been digging the possibilities from the past posts here but still stuck at the population_template command.

Thank you in advance!

The below is the detailed output of the command using a subset of n=24.

population_template ./fod_input -mask_dir ./mask_input wmfod_template.mif -nthread 4 -voxel_size 1.25
population_template: Generating a population-average template from 24 input images
population_template: SH Series detected, performing FOD registration in contrast: fod_input
population_template: ------------------------------------------------------------
population_template: initial alignment of images: mass
population_template: ------------------------------------------------------------
population_template: ------------------------------------------------------------
population_template: linear registration stages:
population_template: ------------------------------------------------------------
population_template: (00) rigid     scale: 0.3000, niter: 100, lmax: 2
population_template: (01) rigid     scale: 0.4000, niter: 100, lmax: 2
population_template: (02) rigid     scale: 0.6000, niter: 100, lmax: 2
population_template: (03) rigid     scale: 0.8000, niter: 100, lmax: 4
population_template: (04) rigid     scale: 1.0000, niter: 100, lmax: 4
population_template: (05) rigid     scale: 1.0000, niter: 100, lmax: 4
population_template: (06) affine    scale: 0.3000, niter: 500, lmax: 2
population_template: (07) affine    scale: 0.4000, niter: 500, lmax: 2
population_template: (08) affine    scale: 0.6000, niter: 500, lmax: 2
population_template: (09) affine    scale: 0.8000, niter: 500, lmax: 4
population_template: (10) affine    scale: 1.0000, niter: 500, lmax: 4
population_template: (11) affine    scale: 1.0000, niter: 500, lmax: 4
population_template: ------------------------------------------------------------
population_template: nonlinear registration stages:
population_template: ------------------------------------------------------------
population_template: (00) nonlinear scale: 0.3000, niter: 5, lmax: 2
population_template: (01) nonlinear scale: 0.4000, niter: 5, lmax: 2
population_template: (02) nonlinear scale: 0.5000, niter: 5, lmax: 2
population_template: (03) nonlinear scale: 0.6000, niter: 5, lmax: 2
population_template: (04) nonlinear scale: 0.7000, niter: 5, lmax: 2
population_template: (05) nonlinear scale: 0.8000, niter: 5, lmax: 2
population_template: (06) nonlinear scale: 0.9000, niter: 5, lmax: 2
population_template: (07) nonlinear scale: 1.0000, niter: 5, lmax: 2
population_template: (08) nonlinear scale: 1.0000, niter: 5, lmax: 4
population_template: (09) nonlinear scale: 1.0000, niter: 5, lmax: 4
population_template: (10) nonlinear scale: 1.0000, niter: 5, lmax: 4
population_template: (11) nonlinear scale: 1.0000, niter: 5, lmax: 4
population_template: (12) nonlinear scale: 1.0000, niter: 5, lmax: 4
population_template: (13) nonlinear scale: 1.0000, niter: 5, lmax: 4
population_template: (14) nonlinear scale: 1.0000, niter: 5, lmax: 4
population_template: (15) nonlinear scale: 1.0000, niter: 5, lmax: 4
population_template: ------------------------------------------------------------
population_template: input images:
population_template: ------------------------------------------------------------
population_template: 	input: CON2001, _c0: "MRCON2001_v1.mif", mask: MRCON2001_v1.mif
population_template: 	input: CON2002, _c0: "MRCON2002_v1.mif", mask: MRCON2002_v1.mif
population_template: 	input: CON2005, _c0: "MRCON2005_v1.mif", mask: MRCON2005_v1.mif
population_template: 	input: CON2009, _c0: "MRCON2009_v1.mif", mask: MRCON2009_v1.mif
population_template: 	input: CON2012, _c0: "MRCON2012_v1.mif", mask: MRCON2012_v1.mif
population_template: 	input: CON2013, _c0: "MRCON2013_v1.mif", mask: MRCON2013_v1.mif
population_template: 	input: TBI3001, _c0: "MRTBI3001_v1.mif", mask: MRTBI3001_v1.mif
population_template: 	input: TBI3007, _c0: "MRTBI3007_v1.mif", mask: MRTBI3007_v1.mif
population_template: 	input: TBI3009, _c0: "MRTBI3009_v1.mif", mask: MRTBI3009_v1.mif
population_template: 	input: TBI3016, _c0: "MRTBI3016_v1.mif", mask: MRTBI3016_v1.mif
population_template: 	input: TBI3017, _c0: "MRTBI3017_v1.mif", mask: MRTBI3017_v1.mif
population_template: 	input: TBI3018, _c0: "MRTBI3018_v1.mif", mask: MRTBI3018_v1.mif
population_template: 	input: TBI3024, _c0: "MRTBI3024_v1.mif", mask: MRTBI3024_v1.mif
population_template: 	input: TBI3025, _c0: "MRTBI3025_v1.mif", mask: MRTBI3025_v1.mif
population_template: 	input: TBI3028, _c0: "MRTBI3028_v1.mif", mask: MRTBI3028_v1.mif
population_template: 	input: TBI3029, _c0: "MRTBI3029_v1.mif", mask: MRTBI3029_v1.mif
population_template: 	input: TBI3031, _c0: "MRTBI3031_v1.mif", mask: MRTBI3031_v1.mif
population_template: 	input: TBI3032, _c0: "MRTBI3032_v1.mif", mask: MRTBI3032_v1.mif
population_template: 	input: TBI3033, _c0: "MRTBI3033_v1.mif", mask: MRTBI3033_v1.mif
population_template: 	input: TBI3037, _c0: "MRTBI3037_v1.mif", mask: MRTBI3037_v1.mif
population_template: 	input: TBI4001, _c0: "MRTBI4001_v1.mif", mask: MRTBI4001_v1.mif
population_template: 	input: TBI4002, _c0: "MRTBI4002_v1.mif", mask: MRTBI4002_v1.mif
population_template: 	input: TBI4003, _c0: "MRTBI4003_v1.mif", mask: MRTBI4003_v1.mif
population_template: 	input: TBI4004, _c0: "MRTBI4004_v1.mif", mask: MRTBI4004_v1.mif
population_template: Generated scratch directory: /media/mcair_store/TBI2018/MR2TBI/Diffusion/fba/mtcsd/template_test/population_template-tmp-PKU6IF/
population_template: Changing to scratch directory (/media/mcair_store/TBI2018/MR2TBI/Diffusion/fba/mtcsd/template_test/population_template-tmp-PKU6IF/)
population_template: Generating initial template
Command:  mraverageheader -fill [/media/mcair_store/TBI2018/MR2TBI/Diffusion/fba/mtcsd/template_test/fod_input/MR*_v1.mif (24 items)] - | mrgrid - regrid -voxel 1.25,1.25,1.25 average_header.mif
population_template: [100%] Importing input masks to average space for template cropping
Command:  mrmath [mask_transformed/*.mif (24 items)] max mask_initial.mif
Command:  mrgrid average_header.mif crop -mask mask_initial.mif average_header_cropped.mif
Function: posix.remove('mask_initial.mif')
Function: posix.remove('average_header.mif')
Function: shutil.move('average_header_cropped.mif', 'average_header.mif')
population_template: [100%] Erasing temporary mask images
Function: shutil.move('average_header.mif', 'average_header_c0.mif')
population_template: [100%] Performing initial rigid registration to template
Command:  mrgrid average_header_tight.mif pad -uniform 10 - | mrgrid - regrid -voxel 1.25,1.25,1.25 average_header.mif
Function: posix.remove('average_header_tight.mif')
population_template: [100%] Reslicing input masks to average header
Command:  mrmath [mask_transformed/*.mif (24 items)] max mask_translated.mif
Command:  mrgrid average_header.mif crop -mask mask_translated.mif average_header_cropped.mif
Command:  mrgrid average_header_cropped.mif pad -uniform 10 average_header.mif
Function: posix.remove('average_header_cropped.mif')
population_template: [100%] Reslicing masks to new padded average header
Function: posix.remove('mask_translated.mif')
population_template: [100%] Reslicing input images to average header
Command:  mrmath [input_transformed_c0/*.mif (24 items)] mean -keep_unary_axes initial_template_c0.mif
population_template: [100%] Optimising template with linear registration (stage 12 of 12; affine)
Command:  mrmath [mask_transformed/*.mif (24 items)] min - | maskfilter - median - | maskfilter - dilate -npass 5 init_nl_template_mask.mif
population_template: [ 10%] Optimising template with non-linear registration (stage 2 of 16)... 
population_template: [ERROR] mrregister /media/mcair_store/TBI2018/MR2TBI/Diffusion/fba/mtcsd/template_test/fod_input/MRTBI3031_v1.mif nl_template00_c0.mif -type nonlinear -nl_niter 5 -nl_warp_full warps_01/TBI3031.mif -transformed input_transformed_c0/TBI3031.mif  -nl_update_smooth 2.0 -nl_disp_smooth 1.0 -nl_grad_step 0.5 -nl_init warps_00/TBI3031.mif -mask1 /media/mcair_store/TBI2018/MR2TBI/Diffusion/fba/mtcsd/template_test/mask_input/MRTBI3031_v1.mif -mask2 nl_template_mask0.mif -datatype float32 -nan -nl_lmax 2 (population_template:1318)
population_template: [ERROR] Information from failed command:
population_template:
                     mrregister: [WARNING] existing output files will be overwritten
                     mrregister: SH image input pair /media/mcair_store/TBI2018/MR2TBI/Diffusion/fba/mtcsd/template_test/fod_input/MRTBI3031_v1.mif, nl_template00_c0.mif
                     mrregister: performing FOD registration
                     mrregister: [ERROR] Permission denied
                     mrregister: [ERROR] error opening image "warps_00/TBI3031.mif"
population_template:
population_template: [ERROR] For debugging, inspect contents of scratch directory: /media/mcair_store/TBI2018/MR2TBI/Diffusion/fba/mtcsd/template_test/population_template-tmp-PKU6IF/
1 Like
2

Hi @lijiang,

Since you’ve got more information here than in the othrer thread, I’ve closed that one off, though much of my suggestions there still apply. But please don’t generate duplicate threads; it won’t actually reduce the time taken for a response.

I presume that /media/mcair_store/ definitely corresponds to the disk with 64TB free? If you’d instructed MRtrix3 to make the script’s scratch directory in /tmp, I could foresee it being a storage space issue, but that doesn’t seem to be the case.

In part this question relates to the “Permission denied” error message. This isn’t wholly generated by MRtrix3, it’s generated by the system and MRtrix3 simply echoes it up to the terminal; and sometimes with deep system messages like this they can be erroneously specific.

You could also try monitoring disk & RAM usage while the script is executing to see if anything is maxxing out. Though given the scratch directory is retained after the failure, the available disk space would be expected to be zero at that point.

There’s nothing particularly special about the number 24, I would not expect it to be a programming limitation of some sort. So beyond that I’m out of ideas.

Rob

3

Is there a disk quota limit set up for this device that you might be exceeding? Is /media/mcair_store/ a local disk or is it a network mounted device?

4

Hi Max, It is a network mounted device.

LI

5

I don’t know whether this is the case here but I’ve seen similar issues with a Samba share that has a compressed file system where I could trigger permission errors with rapid creation and subsequent modification of files.

If possible I’d suggest you use a local drive as a scratch directory to avoid this issue and to speed up the template creation.

6

Thank you very much, Max and Rob.

I tried to use my local drive to run the population_template command and it went smoothly.

Li