Hi Folks
Another beginner question…
I am having trouble using the piping command.
I have two guesses, either I am using some wrong character, OR it could be that on the cluster I am using (m3.massive.org.au) the pipe command tries to place it’s temp file in a location it’s not allowed.
Could that be possible?
M.
I am no expert in regard to any of this, but I would look into building singularity containers to run your pipelines (no pun intended). Gets rid of a lot of this kind of heart ache when running on clusters.
I also found that there is an option to set the location of the temp dir
https://mrtrix.readthedocs.io/en/latest/reference/environment_variables.html#envvar-MRTRIX_TMPFILE_DIR
so I tried
export MRTRIX_TMPFILE_DIR=/scratch/db18/temp
which I know exists and allows me access, but the exact same error message.
But checking the temp dir
[mschira@m3-login2 temp]$ ls
mrtrix-tmp-rbNkbG.mif
finds a temp file. So that means the problem is not before the pipe but - as the error message suggests on reading again - the second half of the command, the mrmath side…
Dunno a singularity container would help. Not if I am doing a stupid spelling error or something.
M.
Oh, and to test the second half of the command I tried
[mschira@m3-login2 temp]$ mrmath mrtrix-tmp-rbNkbG.mif mean mean_b0_PA.mif -axis 3
… which worked like a charm.
It’s just weird.
M.
Yup exceeded my knowledge - hope the overlords chime in.
Hard to tell from the screenshot, but I think it’s the dash issue again… If you copy/paste the command in here as text (formatted using the code tags), we can actually verify on our systems.
I’ve done a little digging on the dash issue yesterday, and I think I’ve got a ‘fix’ for the option parsing side of things. But as mentioned in the issue description, piping might also need to be fixed independently…
Yea, no, it’s not. I have made very sure it is not.
Besides as I pointed out I can copy the second half of the argument and enter the temp file saved by the broken pipe and it works.
I am beginning to suspect there is something seriously wrong with the shell.
I am struggling with dwifslprepro, too. It works most of the time but trying to use eddy_options is driving me nuts.
works:
dwifslpreproc dwi_den_unr.mif dwi_den_unr_preproc.mif -pe_dir AP -rpe_pair -se_epi b0_pair.mif
breaks straight away:
dwifslpreproc dwi_den_unr.mif dwi_den_unr_preproc.mif -pe_dir AP -rpe_pair -se_epi b0_pair.mif -eddy_options "--slm=linear"
dwifslpreproc complains:
Error: argument -eddy_options: expected one argument
breaks later…:
dwifslpreproc dwi_den_unr.mif dwi_den_unr_preproc.mif -pe_dir AP -rpe_pair -se_epi b0_pair.mif -eddy_options slm=linear
specifically: proceeds to eddy, which then complains
slm=linear: is an unrecognised token!
It kinda makes sense, eddy should get the parameter --slm=linear.
I have tried all versions of " and ’ - – and whatever I can think of. I am going to campus and going to install the stuff on one of our computers.
It’s not like I am computer illiterate either. I am quite successful getting ANTs to work…
M.
It appears your quotation marks are not ASCII format, you’re using a dash instead of - and the leading space is missing.
dwifslpreproc dwi_den_unr.mif dwi_den_unr_preproc.mif -pe_dir AP -rpe_pair -se_epi b0_pair.mif -eddy_options " --slm=linear"
I suggest you try to figure out if and when - becomes – for instance with this UTF to ascii converter.
Actually, I think that was fine – the raw markdown is OK, it’s just that it gets pretty-formatted by the forum software (Discourse): -- becomes a long dash ‘–’, and "quote" become “quotes”…
Which emphasises the need to use the appropriate markdown for verbatim quoting… Use the preformatted text button in the toolbar above to do that:
It’ll avoid confusion down the track… I’ve edited your previous post to use that.
So I think the issue is indeed deeper than that…
I note there’s a few things happening before the command runs, including setting up VGLrun and a mount point. I’ve never seen this before, it could have something to do with it…? Can you clarify what’s going on here?
That point is probably the reason for the dwifslpreproc issue though: as mentioned in the docs, there needs to be a space in the argument for that option to work. I’ll freely admit it’s not obvious, and maybe this is something we could find a better way to support…
OK, so that means the first command ran fine, so that first dash must have been a regular ASCII dash. All it does is print the filename of the temporary file to stdout, so you could just intercept that using command substitution:
TMPFILE=$(mrconvert b0_PA/ -)
then verify that you actually got a sensible filename in the expected location:
echo $TMPFILE
then try piping that into the next stage:
echo $TMPFILE | mrmath - mean mean_b0_PA.mif -axis 3
If that still doesn’t work, then there’s something really odd going on…
By the way, if the command you’re struggling with was copy/pasted from the BATMAN tutorial (page 8), then it will be a Unicode dash issue:
If I copy/paste that here, I get:
mrconvert b0_PA/ - | mrmath – mean mean_b0_PA.mif –axis 3
and you’ll note the first dash is OK, but not the one immediately after mrmath, or the one at the start of the -axis option. I think we’ll need to fix that, it seems to cause a lot of confusion…
For reference, the correct version is:
mrconvert b0_PA/ - | mrmath - mean mean_b0_PA.mif -axis 3
Actually, I think that was fine – the raw markdown is OK, it’s just that it gets pretty-formatted by the forum software (Discourse): – becomes a long dash ‘–’, and “quote” become “quotes”…
Ha, the typographer tripped me up. While code blocks are the solution, maybe worth considering switching it off anyway for quotation marks or all together for the community forum?
Almost solved issue no one (piping). Thanks!
Indeed the output to stdout the is broken, it contains that vgl error message first, then the filename. So yea, that explains why the pipe does not work.
I’ll see what I can do about that…
Now with respect to the preproc issue… Could be somehow related? I dunno?
And yes indeed those are taken from Robin’s tutorial. err Batman.
And yes, after this tripped me with wrong dash once, I payed attention and essentially typed EVERY dash myself. There are quite a few proper errors in that tutorial btw. like arguments in the wrong order. Not sure how that happened. I am marking them in the pdf as I go through it.
And yea, I tried with and without space. With space first, as that is what the examples use…
And with respect to non ascii ", I am typing it with my keyboard, hitting shift and two keys right from L.
So I guess it should be ascii?
and I tried one, two minus signs, zero, one and two spaces; ’ ', " " and ``.
Every single time the preproc script complains it did not get an argument.
And as I said I first wrote the command without eddy_options, it worked, then removed the output file, hit the upwards key and then manually type the eddy_options.You know just to be sure I am not having any weird letters.That’s how paranoid I have become 
Unless there is something weird with the keyboard drivers?
Though I don’t think so, I am having the same issues when using the mac shell and -ssh and when using a virtual desktop.
OK, that explains things. If you can modify whatever wrapper script or alias is being invoked here to produce its output to stderr, that would fix the issue. We reserve stdout for pipe-able outputs only, which I think (hope?) is relatively standard Unix practice.
Thanks, we’d appreciate the feedback on that front. I expect a few of those will be due to the fact that the tutorial was written a few versions ago, things will have changed somewhat. But the dash issue has been problematic. We’re currently talking with @martahedl to update the tutorial, so hopefully this will all be fixed up eventually.
OK, so something else must indeed be going on. I would suspect your environment might be causing trouble here. Would be worth running with the -debug and -nocleanup options to get more verbose debugging output and to preserve all intermediate files, including the full command log. That will hopefully give us a clearer insight into what’s going on there.
Yes, I think that was a red herring, don’t worry about that… I don’t think your keyboard is to blame!
Yes. I’ll sort out that [VGL… error thing first, then see what other problems that might solve.
Interestingly, that [VGL … thing is limited to the mrtrix module. But to all of them. fsl does not have it, or ls or others.
i.e.
ls | less
work fine with no [vgl …
I am super surprised that it almost seems like I am the first having issues with mrtrix on M3.
Been having nothing but great experiences with M3 so far, but atm it seems to be causing me a lot of grief.
I much appreciate the help though.
But for today I am gonna hit the pause. Good timing to do on a positive note.
Cheers
M.
I’m guessing whoever installed MRtrix on that system used some symbolic link trickery1 to wrap all our commands to set up VirtualGL (one of the few ways to get mrview & shview to work over remote X11 connections). But there’s no point doing that for any of our commands other than mrview & shview…
Good luck in any case…
1for example, all commands are symbolic links to the same wrapper script, which executes the command actually desired by inspecting its argv[0] – the command as typed by the user – and calling that from the known location of the real executable (after having performed whatever setup was required). This is used quite often in various projects.
Right.
Let me hazard a wild guess: said symbolic link trickery could mess up my quotation marks so that they never arrive at the wrapper script.
Then of course dwifslpreproc would complain about missing an argument.
M.
not enormously unlikely… It should be relatively easy to avoid that, but that’s up to whoever wrote that script…
That point is probably the reason for the
dwifslpreprocissue though: as mentioned in the docs, there needs to be a space in the argument for that option to work. I’ll freely admit it’s not obvious, and maybe this is something we could find a better way to support…
Unfortunately the problem here arises in the Python argparse module upstream of MRtrix3 handling. If there’s no space within the quotation marks, the whole argument just disappears; and if there’s multiple entries space-delimited within the quotation marks, if there’s no extraneous space at the start or end of the string, then one of the entries within the quotation marks gets dropped. I’m kind of surprised that such a bug can still persist TBH. Unless it’s fixed upstream, it contributes along with other issues to the need to implement MRtrix3 Python command-line parsing from scratch… Any takers? 