How to exclude a folder/file in foreach

1

Hello all,

How can I exclude some folders/files when I use the foreach command?

Thank you,
Mahmoud

2

Could you not use a regular for-loop to write the file/folder paths to a txt file and then use that text file in your foreach statement? Perhaps not elegant but it would work.

3

Simplest is to use pattern matching or some other shell substitution – if that’s enough for your needs (depends what you’re trying exclude). Another option is to write the files you’re interested in to file, then pass that list to foreach. Yet another option is use symbolic links (see below). These are all facilities provided by the shell and/or the OS, by the way – nothing to do with MRtrix3 as such…

Using shell expansion

The foreach command expects a list of inputs before the colon. Typically that’s provided as a pattern to be expanded by the shell, e.g.:

foreach inputs/* : command IN outputs/NAME

But what actually happens is that the shell (e.g. bash) will expand it for you, so in fact, if the folder inputs contains the following contents:

ls inputs/
cont10.mif  cont12.mif  cont14.mif  cont16.mif  cont18.mif  cont2.mif  cont4.mif  cont6.mif  cont8.mif  pat10.mif  pat12.mif  pat14.mif  pat16.mif  pat18.mif  pat1.mif   pat21.mif  pat2.mif  pat4.mif  pat6.mif  pat8.mif
cont11.mif  cont13.mif  cont15.mif  cont17.mif  cont1.mif   cont3.mif  cont5.mif  cont7.mif  cont9.mif  pat11.mif  pat13.mif  pat15.mif  pat17.mif  pat19.mif  pat20.mif  pat22.mif  pat3.mif  pat5.mif  pat7.mif  pat9.mif

then the command above gets expanded by the shell to:

foreach inputs/cont10.mif inputs/cont11.mif inputs/cont12.mif inputs/cont13.mif inputs/cont14.mif inputs/cont15.mif inputs/cont16.mif inputs/cont17.mif inputs/cont18.mif inputs/cont1.mif inputs/cont2.mif inputs/cont3.mif inputs/cont4.mif inputs/cont5.mif inputs/cont6.mif inputs/cont7.mif inputs/cont8.mif inputs/cont9.mif inputs/pat10.mif inputs/pat11.mif inputs/pat12.mif inputs/pat13.mif inputs/pat14.mif inputs/pat15.mif inputs/pat16.mif inputs/pat17.mif inputs/pat18.mif inputs/pat19.mif inputs/pat1.mif inputs/pat20.mif inputs/pat21.mif inputs/pat22.mif inputs/pat2.mif inputs/pat3.mif inputs/pat4.mif inputs/pat5.mif inputs/pat6.mif inputs/pat7.mif inputs/pat8.mif inputs/pat9.mif : command IN outputs/NAME

And this happens before the foreach is itself invoked. You can verify this yourself simply by adding echo at the beginning of the command.

This means there’s nothing stopping you from explicitly listing the files you’re interested in manually – though it might admittedly be a little tedious to do that. What might work better is to use one of the many shortcuts that the shell provides for you, for example:

  • foreach inputs/pat*.mif : command IN outputs/NAME

    will match all files named inputs/pat*.mif, where * means ‘any number of characters’ – so files that start in pat and end in .mif.

  • foreach inputs/pat?.mif : command IN outputs/NAME

    will match all files named inputs/pat?.mif, where * means ‘any one character’ – so that expands to files pat1.mif through to pat9.mif.

  • foreach inputs/cont{1..5}.mif : command IN outputs/NAME

    will expand to files cont1.mif through to cont5.mif.

  • foreach inputs/cont{{1..5},{12..18}}.mif : command IN outputs/NAME

    will expand to files cont1.mif through to cont5.mif and cont12.mif through to cont18.mif.

  • If the files in inputs/ were numbered as pat01.mif, pat02.mif, … , pat22.mif, then:

    foreach inputs/pat{01..15}.mif : command IN outputs/NAME

    will expand to files pat01.mif through to pat15.mif. Useful if you have leading zeros in your numbering.

  • a more explicit listing, but better than manually entering the entire filename for each one:

    foreach inputs/{pat1,pat5,cont12,cont17}.mif : command IN outputs/NAME

    will expand to the files pat1.mif, pat5.mif, cont12.mif & cont17.mif.

Passing a file to foreach

If you write all the files you’re interested in into a file, for example list.txt, with contents:

inputs/cont13.mif
inputs/cont5.mif
inputs/cont6.mif
inputs/cont9.mif
inputs/pat14.mif
inputs/pat1.mif
inputs/pat20.mif
inputs/pat3.mif

then you can pass that to foreach like this:

foreach $(cat list.txt) : command IN outputs/NAME

Hint: you can generate your list.txt file quite easily with a bit of shell redirection:

ls inputs/* > list.txt

then open the newly-created file list.txt in a text editor and remove the lines you want to exclude.

Using symbolic links

If that’s still not enough, an alternative (available on Linux & maxcOS only – not Windows unfortunately) is to use symbolic links. Create another empty folder, then create symbolic links in it that link to the entries you do want to process, then invoke foreach on that. For example:

mkdir selected

# create as many links as required
# (note that you can use the same shortcuts as above):
ln -sr inputs/pat1.mif selected/
ln -sr inputs/pat9.mif selected/ 
ln -sr inputs/cont{3,5,12,15}.mif selected/

# inspect your new folder with selected inputs:
ls -l selected/
lrwxrwxrwx 1 jdt13 perinatal 20 Apr 10 15:01 cont12.mif -> ../inputs/cont12.mif
lrwxrwxrwx 1 jdt13 perinatal 20 Apr 10 15:01 cont15.mif -> ../inputs/cont15.mif
lrwxrwxrwx 1 jdt13 perinatal 19 Apr 10 15:01 cont3.mif -> ../inputs/cont3.mif
lrwxrwxrwx 1 jdt13 perinatal 19 Apr 10 15:01 cont5.mif -> ../inputs/cont5.mif
lrwxrwxrwx 1 jdt13 perinatal 18 Apr 10 15:01 pat1.mif -> ../inputs/pat1.mif
lrwxrwxrwx 1 jdt13 perinatal 18 Apr 10 15:01 pat9.mif -> ../inputs/pat9.mif

# run your foreach command on these inputs:
foreach selected/* : command IN outputs/NAME

The advantage there is that you can use this selection again very easily, and it doesn’t take up any extra storage space.

There’s probably other ways to do this that I haven’t thought of, but that should cover most situations…

2 Likes
4

This might be a very useful post/doc for our FAQ actually!