BIDS App Singularity container: "Exec format error"

fmriguy · March 1, 2019, 6:35pm

I apologize for posting here since I am unable to post a new thread for some reason (perhaps my account is new).

I just started exploring the BIDS app for MRtrix3_connectome, downloaded a singularity image for it, and tried to use the following commands to get it running:

unset PYTHONPATH
export PYTHONNOUSERSITE=1
dataDIR=/PROJECTS/data_PILOT
sing_img=/PROJECTS/singularity/mrtrix3_connectome.simg

module load singularity

cd $dataDIR

singularity run -B /PROJECTS:/PROJECTS $sing_img $dataDIR $dataDIR/output participant --participant_label sub-pilot20190225 --parcellation desikan

This is a BIDS validated dataset, however, I am getting the following
error:/.singularity.d/runscript: 3: exec: /mrtrix3_connectome.py: Exec format error

Does anyone know how this can be fixed? I am sorry but I am very new to MRtrix3. Thanks for your help.

rsmith · March 3, 2019, 7:10am

Hi @fmriguy,

From memory we had to forbid brand new accounts from creating new topics in order to prevent spambots. I’ve moved it to a new topic for you, and you should be free to do such from now on

I don’t actually (yet) provide the requisite file for building a Singularity image for this tool. Either you’ve used the docker2singularity conversion tool (which I’ve not used myself for a long time), or the location where you’ve accessed the container is in fact a fork of my own work and so I can’t really help a great deal. It seems to relate to what’s referred to in Docker speak as the “entrypoint”, seemingly trying to interpret that file as a binary executable when it is in fact a Python script. Normally this is identified based on the shebang at the head of the file, but perhaps Singularity requires this to be more explicit during container production?

@lvinokur: Ever encountered this one?

Rob

lvinokur · March 3, 2019, 11:32pm

Hi @fmriguy ,
I complete second what @rsmith said and will just add some details :
First of all - note you are downloading a docker container, which needs to be explicitly converted to singularity. It’s a simple command that requires you use docker2singularity.
If you’ve done that already and docker2singularity didn’t raise any errors then I have two suggestions:

Use singularity run rather than singularity exec
Check the python shebang. At one time is has given me similar woes. Even a blank line before the shebang can cause this error.
In either case - let us know if either of these sort the problem

Lea

fmriguy · March 7, 2019, 4:14pm

Thank you for your replies and help @rsmith and @lvinokur

I was wondering if the mrtrix3_connectome BIDS app can handle custom parcellations beyond the few that are already built in. For instance, the Schaefer et al (2018) parcellations from Thomas Yeo’s group are gaining popularity, and it would be nice to be able to feed in the GM roi’s from the Schaefer parcellation and obtain mrtrix3 connectomes.

rsmith · March 17, 2019, 6:08am

There’s no way to handle parcellations outside of the set currently supported by the script that does not require explicit manipulation of the code. You can however make requests for such by listing it as a GitHub issue; or fork the repository and make the changes yourself.

For the parcellations you describe, I’ve got a funny feeling that I’ve already implemented those somewhere; I’m just having a lot of trouble finding time to put into this tool. I’ll see if I can find it…

Mike · July 3, 2019, 9:18am

Dear @lvinokur and @rsmith,

would you mind sharing the docker2singularity command you are using? I tried the one from the BIDS Apps Gorgolewski et al. paper

docker run --privileged -ti --rm
-v /var/run/docker.sock:/var/run/docker.sock
-v D:\singularity_images:/output
singularityware/docker2singularity
bids/example:0.0.4

But it gives me this:

sh-3.2# docker run --privileged -ti --rm -v /var/run/docker.sock:/var/run/docker.sock -v  /Users/user/Desktop/Pipeline_Docker:/output singularityware/docker2singularity bids/mrtrix3_connectome

Image Format: squashfs
Docker Image: bids/mrtrix3_connectome

Inspected Size: 18536 MB

(1/10) Creating a build sandbox...
(2/10) Exporting filesystem...
(3/10) Creating labels...
(4/10) Adding run script...
(5/10) Setting ENV variables...
(6/10) Adding mount points...
(7/10) Fixing permissions...
(8/10) Stopping and removing the container...
(9/10) Building squashfs container...
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
FATAL:   While performing build: packer failed to pack: cp Failed: exit status 1: cp: write error: No space left on device
cp: write error: No space left on device
cp: write error: No space left on device
cp: write error: No space left on device
cp: write error: No space left on device
cp: write error: No space left on device
cp: can't create '/tmp/sbuild-784341039/fs/./opt/freesurfer/bin/mri_fuse_segmentations': No space left on device
<several dozen cp-write errors snipped>

Also, would you mind sharing the specific commands you use for running it?

Many thanks!!
Best,
Micha

dkp · July 3, 2019, 2:08pm

Singularity is a fast moving target. The modern version builds sif files. Does docker2singularity even work anymore?
Here’s what I run (assumes you are on a machine with singularity):

singularity build mrtrix3_connectome.sif docker://bids/mrtrix3_connectome

But note you are running out of space (I did too). Singularity fills up these hidden dirs in your home area. I moved .singularity/docker and .singularity/cache to somewhere I have more GB of space (not my home dir on the HPC)

And I added these to .bash_profile:

export SINGULARITY_PULLFOLDER=/extra/dkp/build_sif/docker

export SINGULARITY_CACHEDIR=/extra/dkp/build_sif/cache

I presume docker2singularity has some similar space issue with hidden docker dirs, but you might go around the problem with a solution like I describe.

Mike · July 4, 2019, 8:55am

Many thanks @dkp! It worked to built the sif. I think the running-out-of-space-error was not referring to my local filesystem – there is 1 TB of free space – I think it referred to the inside of the container. Anyway, thanks again, will try to run it now.

rsmith · July 11, 2019, 12:11pm

I recently got native Singularity building working for this script, which should mean I can get it hosted on SingularityHub, at which point Docker shouldn’t be required at all. I’ll see about packaging that up soon. I was hoping to get CUDA working in Singularity as well (for eddy), but no success yet on that front.

dkp · July 14, 2019, 5:09pm

Here’s my earlier work on getting cuda tools to run on our HPC (it is the older FSL with bedpostx, probtrackx and eddy cuda downloaded explicitly, but it does work on our HPC (at least the last time I tried it with singularity 2.x.x, and explicit testing on probtrackx and bedpostx) https://bitbucket.org/dpat/bipbids/src/master/Singularity

I have not tried it with the newest fsl and singularity. I depended heavily on the hard work of a colleague, Chidi Ugonna.

-D

cchin · January 6, 2021, 5:24pm

Hi Robert, by chance were you able to find your script for the schaefer parcellation? Thanks.

rsmith · January 21, 2021, 5:33am

Hi Cherise,

No, the Schaefer parcellations didn’t end up getting included in the 0.5.0 update of that tool. It’s in the issue list, but unfortunately I have other expectations and projects that are taking precedence right now. Having said that, it would not take a high level of expertise in Python if anyone wanted to make an attempt at integrating new parcellations.

Rob