Issues running `dwi2response` on cluster

1

Hi! I’m trying to run a tractography script on a HPC environment and am having trouble when submitting jobs with one of the mrtrix functions, dwi2response, and subfunction mrinfo which tries to write to a directory it shouldn’t. My script runs fine when run on development nodes, but when submitted as job it fails. Here are the details:

  • running through singularity container (I also did with conda directly, same issues)

    singularity container def file 
    Bootstrap: docker
From: continuumio/miniconda3:latest

%post
    conda update -n base -c defaults conda
    conda install -c mrtrix3 mrtrix3

%runscript
    exec "$@"

  • Call to dwi2response:

    mrtrixcall="singularity exec \
  --env MRTRIX_NTHREADS=${MRTRIX_NTHREADS} \
  --env JOB_SCRIPT=${MRTRIX_TMPFILE_DIR} \             # here I tried with, wihtout, setting JOB_SCRIPT="", to no avail
  --env MRTRIX_TMPFILE_DIR=${MRTRIX_TMPFILE_DIR} \     # subdirectory of $TMPDIR
  --bind ${bids_dir}:${bids_dir} \
  --bind ${TMPDIR}:${TMPDIR} \
  /path/to/mrtrix3.sif"
${mrtrixcall} dwi2response dhollander ${dwi_fp}.mif ${force_flag} \
    ${mrtrix_subjsess_dir}/sub-${subjectID}_ses-${sessionID}_response-wm_dwi.txt \
    ${mrtrix_subjsess_dir}/sub-${subjectID}_ses-${sessionID}_response-gm_dwi.txt \
    ${mrtrix_subjsess_dir}/sub-${subjectID}_ses-${sessionID}_response-csf_dwi.txt

  • output:

    dwi2response: 
dwi2response: Note that this script makes use of commands / algorithms that have relevant articles for citation. Please consult the help page (-help option) for more information.
dwi2response: 
mrinfo: [ERROR] error opening output file "/var/spool/sge/wynton/qb3-id240/job_scripts/mrtrix-tmp-gfMv1g.json": Permission denied

    this last path corresponds to the environment variable $JOB_SCRIPT (more precisely its parent directory), but talking with the cluster admins, we don’t see what business mrinfo has with it.

Any idea what’s going on or how to solve this?
Thanks!

2

Happy New Year everyone! a quick follow up here, anyone has any idea?

3

Hey Pierre - a few random, disjointed thoughts:

  • Do you get any extra info from throwing the -debug flag onto dwi2response?
  • What’s the purpose of the JOB_SCRIPT env var, and why are you setting it to the same location as MRTRIX_TMPFILE_DIR env var?
  • Have you shelled into the singularity container with the same --env and --bind flags and checked what’s in $TMPDIR and $MRTRIX_TMPFILE_DIR? Some issues I’ve had in the past have centered around how my cluster handled tmp directories. Is it possible that they contain two different paths, and then, the mrinfo permission error is because you haven’t actually bound the location of $MRTRIX_TMPFILE_DIR?
  • mrinfo may be used internally to dwi2response for extracting relevant info from the *.mif file and writing it to a temp directory for use downstream, but those files disappear after completion unless you tell the command to leave behind intermediate files. AFAIK, MRtrix3 understands the MRTRIX_TMPFILE_DIR as your desired location for those files, which could be part of why you see that error.

Hope that helps.

4

Thanks for the reply! I delved a bit in the mrtrix code and I think it’s a bug, where dwi2response doesn’t use the -scratch dir or $MRTRIX_TMPFILE_DIR, and instead uses the local directory (which used to be this non-writable $JOB_SCRIPT directory). I solved it by forcing the script to go to my tmpdir cd $TMPDIR at the very beginning.

cd $TMPDIR
MRTRIX_TMPFILE_DIR=${TMPDIR}/mrtrix
mrtrixcall="singularity exec \
    --env MRTRIX_NTHREADS=${MRTRIX_NTHREADS} \
    --env MRTRIX_TMPFILE_DIR=${MRTRIX_TMPFILE_DIR} \
    --bind ${bids_dir}:${bids_dir} \
    --bind ${TMPDIR}:${TMPDIR} \
    /path/to/mrtrix3.sif"
mkdir -p $MRTRIX_TMPFILE_DIR
${mrtrixcall} dwi2response dhollander ${dwi_fp}.mif ${force_flag} \
    -scratch ${MRTRIX_TMPFILE_DIR} \
    -mask $brainmask_fp \
    ${mrtrix_subjsess_dir}/sub-${subjectID}_ses-${sessionID}_response-wm_dwi.txt \
    ${mrtrix_subjsess_dir}/sub-${subjectID}_ses-${sessionID}_response-gm_dwi.txt \
    ${mrtrix_subjsess_dir}/sub-${subjectID}_ses-${sessionID}_response-csf_dwi.txt
1 Like
5

I believe dwi2response should use the directory passed to the scratch option. If it doesn’t, that might indeed be a bug. It should however not use the $MRTRIX_TMPFILE_DIR environment variable to set the scratch directory. This directory is only used to write temporary files when C++ commands are piped. I believe there is currently no environment variable to control the scratch directory for python scripts such as dwi2response. You can set it in the config file though, through ScriptScratchDir (for intermediate files in scripts, note that these can amount to massive amounts, e.g. for the population_template script) which is a distinct option from TmpFileDir (for temporary files from pipes). It might be relevant to add a corresponding environment variable for the ScriptScratchDir config file option though…

1 Like