Unix /tmp becomes full

1

Hello,

Just to let you know that /tmp becomes full after a day of consistent use, on my machine filling 16GB. It looks like mrtrix does not correctly remove some temporary files when finished. It gives errors like these due to the lack of space in tmp.

dwinormalise: [ERROR] dwiextract /home/tom/.temp_dmri_preprocessing.sh_2020070715141466/4_unwarped.mif -bzero - | mrmath - mean - -axis 3 | mrstats - -mask /home/tom/.temp_dmri_preprocessing.sh_2020070715141466/csf_mask.mif -output median (individual.py:66)
dwinormalise: [ERROR] Information from failed command:
dwinormalise:
dwiextract: extracting volumes… [=================
dwiextract: [SYSTEM FATAL CODE: SIGBUS (7)] Bus error: Accessing invalid address (out of storage space?)
mrmath: [ERROR] no filename supplied to standard input (broken pipe?)
mrmath: [ERROR] error opening image “-”
mrstats: [ERROR] no filename supplied to standard input (broken pipe?)
mrstats: [ERROR] error opening image “-”

Is mrtrix_cleanup the current suggested solution? On a cron job? Will changing the TmpFileDir fix it? Even if I move the TempFileDir, i would not want that space to be eaten eventually.

Two related threads Out of storage space? dwi2response bus error - #15 by jdtournier

Fedora 32

== mrview 3.0.0-33-g5b4aef85-dirty ==
64 bit release version, built Jun 11 2020, using Eigen 3.3.7

2

Thanks for the report, that’s a matter of urgent concern. First off, yes, that’s what the mrtrix_cleanup script is supposed to help deal with. However, it’s probably not a good idea to place it in a cron job, given this warning on its help page:

This script should not be run while other MRtrix3 commands are being executed: it may delete temporary items during operation that may lead to unexpected behaviour.

Otherwise, you can also just run rm /tmp/mrtrix-tmp-* manually, that will also work.

I’d like to figure out what’s going on though. There’s a quite a few safeguards in there to make sure temporary files get deleted, so I’m surprised to see this happen so consistently on your system. Are you able to isolate the problematic command? If you run the following, this will list the last command in each temporary file’s command history, which should give us a clue as to what might be going on:

for n in /tmp/mrtrix-tmp-*; do mrinfo -property command_history $n | tail -n 1; done

Of note, I notice that mrinfo itself doesn’t consume temporary files – commands like one leave a temporary file behind:

mrconvert input.mif - | mrinfo -

I don’t think this was part of the plan… And there’s a good chance something like this might be used in some of our scripts, which could explain the issue… I’ll look into it straight away.

3

I’ve pushed a fix for the ’mrinfo not deleting piped images’ bug - pull request here. I’m not sure whether this will fix your issue without further investigation though. If you’re interested in trying it out, feel free to check out the corresponding fix_mrinfo_non_deletion_of_images branch:

git fetch
git checkout fix_mrinfo_non_deletion_of_images
./build

If you do try it out, it would be really helpful to hear back as to whether this fixes the issue you’re facing!

4

Great, thank you. I think the issue is from dwiextract

[tom@localhost tmp]$ for n in /HDD/tmp/mrtrix-tmp-*; do mrinfo -property command_history $n | tail -n 1; done
dwiextract /home/tom/temp_dke_2020070720373979/5_normalised.mif - -export_grad_fsl /home/tom/temp_dke_2020070720373979/b1000_bvec /home/tom/temp_dke_2020070720373979/b1000_bval -shell 1000  (version=3.0.0-33-g5b4aef85-dirty)
dwiextract /home/tom/temp_dke_2020070717240139/5_normalised.mif - -export_grad_fsl /home/tom/temp_dke_2020070717240139/b1000_bvec /home/tom/temp_dke_2020070717240139/b1000_bval -shell 1000  (version=3.0.0-33-g5b4aef85-dirty)
dwiextract /home/tom/temp_dke_2020070720262595/5_normalised.mif - -export_grad_fsl /home/tom/temp_dke_2020070720262595/b1000_bvec /home/tom/temp_dke_2020070720262595/b1000_bval -shell 1000  (version=3.0.0-33-g5b4aef85-dirty)
dwiextract /home/tom/temp_dke_2020070718574161/5_normalised.mif - -export_grad_fsl /home/tom/temp_dke_2020070718574161/b1000_bvec /home/tom/temp_dke_2020070718574161/b1000_bval -shell 1000  (version=3.0.0-33-g5b4aef85-dirty)
... x156 ...

The temp files being saved are the full extracted shell by dwiextract 98MB each

Ive moved the temp directory which solves my problem

5

Thanks, that’s really helpful. I can’t see this particular command having been issued as part of any of our scripts. Maybe this is part of a script you produced? If so, do you mind sharing the script with us – or at least the relevant part of it…?

6

I think you are right, it was an error in my own script:

dwiextract $temp_local_dir/5_normalised.mif - -export_grad_fsl $temp_local_dir/b1000_bvec $temp_local_dir/b1000_bval -shell 1000

note the second argument “-” hyphen used instead of a real file name.

I thought this would write out the bvecs/bvals for the b1000 shell, and not write out the image data for the shell itself to a file. It did write out the bvec/bvals but I assumed the “-” would just throw away the image data instead of writing it. Instead, it wrote the image data for the b1000 shell to tmp. Maybe it is not a bug after all but maybe useful to clarify the behaviour of “-” when used in place of filenames especially if it is used not in conjunction with a pipe, although that was probably my inocrrect use anyway.

Cheers
Tom

7

Thanks Tom, that clarifies things. Indeed, if there’s no second command to catch the output of the first, the temporary file produced by the first command won’t be deleted, and will be left on the filesystem. It’s kind of outlined in the documentation, but I fully agree that needs to be made much clearer – I’ll fix that up ASAP.

Otherwise, I’ve been thinking about how to prevent these kinds of problems over the years, but I don’t think I can come up with a solution that simultaneously provides the requisite guarantees, is sufficiently portable across the different platform, and doesn’t prevent legitimate use cases. Feel free to read through my musings on the issue if you’ve got nothing better to do…