# Piping going wrong **URL:** \ **Category:** Uncategorized\ **Created:** [October 8, 2020, 12:08am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163 "2020-10-08T00:08:13Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![mschira](https://community.mrtrix.org/user_avatar/community.mrtrix.org/mschira/32/2917_2.png) [@mschira](https://community.mrtrix.org/u/mschira)\ **Post date:** [October 8, 2020, 12:08am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/1 "2020-10-08T00:08:13Z")
Hi Folks Another beginner question… I am having trouble using the piping command. ![Screen Shot 2020-10-08 at 11.02.51](https://community.mrtrix.org/uploads/default/original/2X/9/94a141be8c8d82e044f5014ce47cdae8f90ac144.png) I have two guesses, either I am using some wrong character, OR it could be that on the cluster I am using (m3.massive.org.au) the pipe command tries to place it’s temp file in a location it’s not allowed. Could that be possible? M. ---
**Author:** ![grahamwjohnson](https://community.mrtrix.org/letter_avatar_proxy/v4/letter/g/d07c76/32.png) [@grahamwjohnson](https://community.mrtrix.org/u/grahamwjohnson)\ **Post date:** [October 8, 2020, 12:45am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/2 "2020-10-08T00:45:08Z")
I am no expert in regard to any of this, but I would look into building singularity containers to run your pipelines (no pun intended). Gets rid of a lot of this kind of heart ache when running on clusters. ---
**Author:** ![mschira](https://community.mrtrix.org/user_avatar/community.mrtrix.org/mschira/32/2917_2.png) [@mschira](https://community.mrtrix.org/u/mschira)\ **Post date:** [October 8, 2020, 2:18am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/3 "2020-10-08T02:18:15Z")
I also found that there is an option to set the location of the temp dir [https://mrtrix.readthedocs.io/en/latest/reference/environment\_variables.html#envvar-MRTRIX\_TMPFILE\_DIR](https://mrtrix.readthedocs.io/en/latest/reference/environment_variables.html#envvar-MRTRIX_TMPFILE_DIR) so I tried export MRTRIX\_TMPFILE\_DIR=/scratch/db18/temp which I know exists and allows me access, but the exact same error message. But checking the temp dir > [mschira@m3-login2 temp]$ ls > mrtrix-tmp-rbNkbG.mif finds a temp file. So that means the problem is not before the pipe but - as the error message suggests on reading again - the second half of the command, the mrmath side… Dunno a singularity container would help. Not if I am doing a stupid spelling error or something. M. ---
**Author:** ![mschira](https://community.mrtrix.org/user_avatar/community.mrtrix.org/mschira/32/2917_2.png) [@mschira](https://community.mrtrix.org/u/mschira)\ **Post date:** [October 8, 2020, 2:23am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/4 "2020-10-08T02:23:29Z")
Oh, and to test the second half of the command I tried ```auto [mschira@m3-login2 temp]$ mrmath mrtrix-tmp-rbNkbG.mif mean mean_b0_PA.mif -axis 3 ``` … which worked like a charm. It’s just weird. M. ---
**Author:** ![grahamwjohnson](https://community.mrtrix.org/letter_avatar_proxy/v4/letter/g/d07c76/32.png) [@grahamwjohnson](https://community.mrtrix.org/u/grahamwjohnson)\ **Post date:** [October 8, 2020, 2:37am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/5 "2020-10-08T02:37:42Z")
Yup exceeded my knowledge - hope the overlords chime in. ---
**Author:** ![jdtournier](https://community.mrtrix.org/user_avatar/community.mrtrix.org/jdtournier/32/2594_2.png) [@jdtournier](https://community.mrtrix.org/u/jdtournier)\ **Post date:** [October 8, 2020, 6:27am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/6 "2020-10-08T06:27:47Z")
Hard to tell from the screenshot, but I think it’s [the dash issue](https://community.mrtrix.org/t/trouble-with-mrmath/4158/2?u=jdtournier) again… If you copy/paste the command in here as text (formatted using the code tags), we can actually verify on our systems. I’ve done a little digging on the dash issue yesterday, and I think I’ve got a [‘fix’ for the option parsing](https://github.com/MRtrix3/mrtrix3/pull/2193) side of things. But as mentioned in the issue description, piping might also need to be fixed independently… ---
**Author:** ![mschira](https://community.mrtrix.org/user_avatar/community.mrtrix.org/mschira/32/2917_2.png) [@mschira](https://community.mrtrix.org/u/mschira)\ **Post date:** [October 8, 2020, 7:59am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/7 "2020-10-08T07:59:04Z")
Yea, no, it’s not. I have made very sure it is not. Besides as I pointed out I can copy the second half of the argument and enter the temp file saved by the broken pipe and it works. I am beginning to suspect there is something seriously wrong with the shell. I am struggling with dwifslprepro, too. It works most of the time but trying to use eddy\_options is driving me nuts. works: ```auto dwifslpreproc dwi_den_unr.mif dwi_den_unr_preproc.mif -pe_dir AP -rpe_pair -se_epi b0_pair.mif ``` breaks straight away: ```auto dwifslpreproc dwi_den_unr.mif dwi_den_unr_preproc.mif -pe_dir AP -rpe_pair -se_epi b0_pair.mif -eddy_options "--slm=linear" dwifslpreproc complains: Error: argument -eddy_options: expected one argument ``` breaks later…: ```auto dwifslpreproc dwi_den_unr.mif dwi_den_unr_preproc.mif -pe_dir AP -rpe_pair -se_epi b0_pair.mif -eddy_options slm=linear ``` specifically: proceeds to eddy, which then complains ```auto slm=linear: is an unrecognised token! ``` It kinda makes sense, eddy should get the parameter `--slm=linear`. I have tried all versions of " and ’ - – and whatever I can think of. I am going to campus and going to install the stuff on one of our computers. It’s not like I am computer illiterate either. I am quite successful getting ANTs to work… M. ---
**Author:** ![maxpietsch](https://community.mrtrix.org/user_avatar/community.mrtrix.org/maxpietsch/32/25_2.png) [@maxpietsch](https://community.mrtrix.org/u/maxpietsch)\ **Post date:** [October 8, 2020, 8:40am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/8 "2020-10-08T08:40:30Z")
> [@mschira](#): > > ```auto > dwifslpreproc dwi_den_unr.mif dwi_den_unr_preproc.mif -pe_dir AP -rpe_pair -se_epi b0_pair.mif -eddy_options “–slm=linear” > > ``` It appears your quotation marks are not ASCII format, you’re using a dash instead of `-` and the leading space is missing. ```auto dwifslpreproc dwi_den_unr.mif dwi_den_unr_preproc.mif -pe_dir AP -rpe_pair -se_epi b0_pair.mif -eddy_options " --slm=linear" ``` I suggest you try to figure out if and when `-` becomes `–` for instance with [this](https://onlineutf8tools.com/convert-utf8-to-ascii) UTF to ascii converter. ---
**Author:** ![jdtournier](https://community.mrtrix.org/user_avatar/community.mrtrix.org/jdtournier/32/2594_2.png) [@jdtournier](https://community.mrtrix.org/u/jdtournier)\ **Post date:** [October 8, 2020, 8:51am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/9 "2020-10-08T08:51:13Z")
> [@maxpietsch](#): > > It appears your quotation marks are not ASCII format, you’re using a dash instead of `-` Actually, I think that was fine – the raw markdown is OK, it’s just that it gets pretty-formatted by the forum software ([Discourse](https://www.discourse.org/)): `--` becomes a long dash ‘–’, and `"quote"` become “quotes”… Which emphasises the need to use the appropriate markdown for verbatim quoting… Use the preformatted text button in the toolbar above to do that: ![Screenshot from 2020-10-08 09-47-47](https://community.mrtrix.org/uploads/default/original/2X/f/f19c0a90e837aace9a7b661de1c15e30f898e237.png) It’ll avoid confusion down the track… I’ve edited your previous post to use that. So I think the issue is indeed deeper than that… I note there’s a few things happening before the command runs, including setting up VGLrun and a mount point. I’ve never seen this before, it could have something to do with it…? Can you clarify what’s going on here? ---
**Author:** ![jdtournier](https://community.mrtrix.org/user_avatar/community.mrtrix.org/jdtournier/32/2594_2.png) [@jdtournier](https://community.mrtrix.org/u/jdtournier)\ **Post date:** [October 8, 2020, 8:58am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/10 "2020-10-08T08:58:24Z")
> [@maxpietsch](#): > > and the leading space is missing. > > ```auto > dwifslpreproc dwi_den_unr.mif dwi_den_unr_preproc.mif -pe_dir AP -rpe_pair -se_epi b0_pair.mif -eddy_options " --slm=linear" > > ``` That point is probably the reason for the `dwifslpreproc` issue though: as [mentioned in the docs](https://mrtrix.readthedocs.io/en/3.0.2/reference/commands/dwifslpreproc.html#options), there needs to be a space in the argument for that option to work. I’ll freely admit it’s not obvious, and maybe this is something we could find a better way to support… ---
**Author:** ![jdtournier](https://community.mrtrix.org/user_avatar/community.mrtrix.org/jdtournier/32/2594_2.png) [@jdtournier](https://community.mrtrix.org/u/jdtournier)\ **Post date:** [October 8, 2020, 9:07am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/11 "2020-10-08T09:07:49Z")
> [@mschira](#): > > Oh, and to test the second half of the command I tried > > ```auto > [mschira@m3-login2 temp]$ mrmath mrtrix-tmp-rbNkbG.mif mean mean_b0_PA.mif -axis 3 > > ``` > > … which worked like a charm. OK, so that means the first command ran fine, so that first dash must have been a regular ASCII dash. All it does is print the filename of the temporary file to stdout, so you could just intercept that using command substitution: ```auto TMPFILE=$(mrconvert b0_PA/ -) ``` then verify that you actually got a sensible filename in the expected location: ```auto echo $TMPFILE ``` then try _piping_ that into the next stage: ```auto echo $TMPFILE | mrmath - mean mean_b0_PA.mif -axis 3 ``` If that _still_ doesn’t work, then there’s something really odd going on… ---
**Author:** ![jdtournier](https://community.mrtrix.org/user_avatar/community.mrtrix.org/jdtournier/32/2594_2.png) [@jdtournier](https://community.mrtrix.org/u/jdtournier)\ **Post date:** [October 8, 2020, 9:18am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/12 "2020-10-08T09:18:59Z")
By the way, if the command you’re struggling with was copy/pasted from the [BATMAN tutorial](https://osf.io/ht7zv/) (page 8), then it will be a Unicode dash issue: ![Screenshot from 2020-10-08 10-15-49](https://community.mrtrix.org/uploads/default/original/2X/0/08272cfdb6bcc88cfb427855caf3f7dd136d092a.png) If I copy/paste that here, I get: ```auto mrconvert b0_PA/ - | mrmath – mean mean_b0_PA.mif –axis 3 ``` and you’ll note the first dash is OK, but not the one immediately after `mrmath`, or the one at the start of the `-axis` option. I think we’ll need to fix that, it seems to cause a lot of confusion… For reference, the correct version is: ```auto mrconvert b0_PA/ - | mrmath - mean mean_b0_PA.mif -axis 3 ``` ---
**Author:** ![maxpietsch](https://community.mrtrix.org/user_avatar/community.mrtrix.org/maxpietsch/32/25_2.png) [@maxpietsch](https://community.mrtrix.org/u/maxpietsch)\ **Post date:** [October 8, 2020, 9:30am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/13 "2020-10-08T09:30:24Z")
> Actually, I think that was fine – the raw markdown is OK, it’s just that it gets pretty-formatted by the forum software (Discourse): – becomes a long dash ‘–’, and “quote” become “quotes”… Ha, the [typographer](https://markdown-it.github.io/) tripped me up. While code blocks are the solution, maybe worth considering switching it [off](https://community.mrtrix.org/admin/site_settings/category/all_results?filter=typogra) anyway for quotation marks or all together for the community forum? ---
**Author:** ![mschira](https://community.mrtrix.org/user_avatar/community.mrtrix.org/mschira/32/2917_2.png) [@mschira](https://community.mrtrix.org/u/mschira)\ **Post date:** [October 8, 2020, 10:12am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/14 "2020-10-08T10:12:30Z")
Almost solved issue no one (piping). Thanks! Indeed the output to stdout the is broken, it contains that vgl error message first, then the filename. So yea, that explains why the pipe does not work. I’ll see what I can do about that… Now with respect to the preproc issue… Could be somehow related? I dunno? And yes indeed those are taken from Robin’s tutorial. err Batman. And yes, after this tripped me with wrong dash once, I payed attention and essentially typed EVERY dash myself. There are quite a few proper errors in that tutorial btw. like arguments in the wrong order. Not sure how that happened. I am marking them in the pdf as I go through it. And yea, I tried with and without space. With space first, as that is what the examples use… And with respect to non ascii ", I am typing it with my keyboard, hitting shift and two keys right from L. So I guess it should be ascii? and I tried one, two minus signs, zero, one and two spaces; ’ ', " " and ``. Every single time the preproc script complains it did not get an argument. And as I said I first wrote the command without eddy\_options, it worked, then removed the output file, hit the upwards key and then manually type the eddy\_options.You know just to be sure I am not having any weird letters.That’s how paranoid I have become 🤯 Unless there is something weird with the keyboard drivers? Though I don’t think so, I am having the same issues when using the mac shell and -ssh and when using a virtual desktop. ---
**Author:** ![jdtournier](https://community.mrtrix.org/user_avatar/community.mrtrix.org/jdtournier/32/2594_2.png) [@jdtournier](https://community.mrtrix.org/u/jdtournier)\ **Post date:** [October 8, 2020, 10:30am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/15 "2020-10-08T10:30:41Z")
> [@mschira](#): > > Indeed the output to stdout the is broken, it contains that vgl error message first, then the filename. So yea, that explains why the pipe does not work. OK, that explains things. If you can modify whatever wrapper script or alias is being invoked here to produce its output to stderr, that would fix the issue. We reserve stdout for pipe-able outputs only, which I _think_ (hope?) is relatively standard Unix practice. > [@mschira](#): > > There are quite a few proper errors in that tutorial btw. like arguments in the wrong order. Not sure how that happened. I am marking them in the pdf as I go through it. Thanks, we’d appreciate the feedback on that front. I expect a few of those will be due to the fact that the tutorial was written a few versions ago, things will have changed somewhat. But the dash issue has been problematic. We’re currently talking with @martahedl to update the tutorial, so hopefully this will all be fixed up eventually. > [@mschira](#): > > And yea, I tried with and without space. With space first, as that is what the examples use… OK, so something else must indeed be going on. I would suspect your environment might be causing trouble here. Would be worth running with the `-debug` and `-nocleanup` options to get more verbose debugging output and to preserve all intermediate files, including the full command log. That will hopefully give us a clearer insight into what’s going on there. > [@mschira](#): > > And with respect to non ascii ", I am typing it with my keyboard, hitting shift and two keys right from L. > So I guess it should be ascii? Yes, I think that was a red herring, don’t worry about that… I don’t think your keyboard is to blame! ---
**Author:** ![mschira](https://community.mrtrix.org/user_avatar/community.mrtrix.org/mschira/32/2917_2.png) [@mschira](https://community.mrtrix.org/u/mschira)\ **Post date:** [October 8, 2020, 10:47am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/16 "2020-10-08T10:47:14Z")
Yes. I’ll sort out that [VGL… error thing first, then see what other problems that might solve. Interestingly, that [VGL … thing is limited to the mrtrix module. But to all of them. fsl does not have it, or ls or others. i.e. > ls | less > work fine with no [vgl … I am super surprised that it almost seems like I am the first having issues with mrtrix on M3. Been having nothing but great experiences with M3 so far, but atm it seems to be causing me a lot of grief. I much appreciate the help though. But for today I am gonna hit the pause. Good timing to do on a positive note. Cheers M. ---
**Author:** ![jdtournier](https://community.mrtrix.org/user_avatar/community.mrtrix.org/jdtournier/32/2594_2.png) [@jdtournier](https://community.mrtrix.org/u/jdtournier)\ **Post date:** [October 8, 2020, 11:34am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/17 "2020-10-08T11:34:06Z")
> [@mschira](#): > > that [VGL … thing is limited to the mrtrix module. But to all of them. fsl does not have it, or ls or others. I’m guessing whoever installed MRtrix on that system used some symbolic link trickery1 to wrap all our commands to set up VirtualGL (one of the few ways to get `mrview` & `shview` to work over remote X11 connections). But there’s no point doing that for any of our commands other than `mrview` & `shview`… Good luck in any case… * * * 1for example, all commands are symbolic links to the same wrapper script, which executes the command actually desired by inspecting its `argv[0]` – the command as typed by the user – and calling that from the known location of the real executable (after having performed whatever setup was required). This is used quite often in various projects. ---
**Author:** ![mschira](https://community.mrtrix.org/user_avatar/community.mrtrix.org/mschira/32/2917_2.png) [@mschira](https://community.mrtrix.org/u/mschira)\ **Post date:** [October 8, 2020, 7:47pm UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/18 "2020-10-08T19:47:36Z")
Right. Let me hazard a wild guess: said symbolic link trickery could mess up my quotation marks so that they never arrive at the wrapper script. Then of course dwifslpreproc would complain about missing an argument. M. ---
**Author:** ![jdtournier](https://community.mrtrix.org/user_avatar/community.mrtrix.org/jdtournier/32/2594_2.png) [@jdtournier](https://community.mrtrix.org/u/jdtournier)\ **Post date:** [October 8, 2020, 7:50pm UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/19 "2020-10-08T19:50:13Z")
😜 not enormously unlikely… It _should_ be relatively easy to avoid that, but that’s up to whoever wrote that script… ---
**Author:** ![rsmith](https://community.mrtrix.org/user_avatar/community.mrtrix.org/rsmith/32/2672_2.png) [@rsmith](https://community.mrtrix.org/u/rsmith)\ **Post date:** [October 9, 2020, 1:29am UTC](https://community.mrtrix.org/t/piping-going-wrong/4163/20 "2020-10-09T01:29:21Z")
> That point is probably the reason for the `dwifslpreproc` issue though: as mentioned in the docs, there needs to be a space in the argument for that option to work. I’ll freely admit it’s not obvious, and maybe this is something we could find a better way to support… Unfortunately the problem here arises in the Python `argparse` module upstream of _MRtrix3_ handling. If there’s no space within the quotation marks, the whole argument just disappears; and if there’s multiple entries space-delimited within the quotation marks, if there’s no extraneous space at the start or end of the string, then one of the entries within the quotation marks gets dropped. I’m kind of surprised that such a bug can still persist TBH. Unless it’s fixed upstream, it contributes along with other issues to the need to [implement _MRtrix3_ Python command-line parsing from scratch](https://github.com/MRtrix3/mrtrix3/issues/1608)… Any takers? 😆 [Next page](https://community.mrtrix.org/t/piping-going-wrong/4163.md?page=2)