Mrtrix redirect output and error to a file

1

Dear all

a new thread for a small question : how to properly redirect mrtrix output command ?
all command line have a nice output in a terminal (with an updated progress) but when I redirect the output to a file it put each progress to the output, which end up with big logfile not very easy to read (150M for a tckgen outpu) :

for instance

9676750 selectedtckgen: [ 96%] 452436141 generated, 9676753 selectedtckgen: [ 96%] 452436371 generated, 9676762 selectedtckgen: [ 96%] 452436491 generated, 9676764 selectedtckgen: [ 96%] 452436641 generated, 9676775 selectedtckgen: [ 96%] 452436831 generated, 9676783 selectedtckgen: [ 96%] 452437031 generated, 9676790 selectedtckgen: [ 96%] 452437201 generated, 9676796 selectedtckgen: [ 96%] 452437411 generated, 9676799 selectedtckgen: [ 96%] 452437561 generated, 9676806 selectedtckgen: [ 96%] 452437701 generated, 9676809 selectedtckgen: [ 96%] 452437871 generated, 9676815 selectedtckgen: [ 96%] 452438041 generated, 9676820 selectedtckgen: [ 96%] 452438181 generated, 9676823 selectedtckgen: [ 96%] 452438351 generated, 9676830 selectedtckgen: [ 96%] 452438511 generated, 9676837 selectedtckgen: [ 96%] 452438651 generated, 9676842 selectedtckgen: [ 96%] 452438771 generated, 9676845 selectedtckgen: [ 96%] 452438971 generated, 9676854 selectedtckgen: [ 96%] 452439161 generated, 9676860 selectedtckgen: [ 96%] 452439361 generated, 9676865 selectedtckgen: [ 96%] 452439531 generated, 9676873 selectedtckgen: [ 96%] 452439681 generated, 9676876 selectedtckgen: [ 96%] 452439841 generated, 9676878 selectedtckgen: [ 96%] 452440021 generated, 9676887 selectedtckgen: [ 96%] 452440191 generated, 9676891 selectedtckgen: [ 96%] 452440341 generated, 9676893 selectedtckgen: [ 96%] 452440501 generated, 9676899 selectedtckgen: [ 96%] 452440711 generated, 9676903 selectedtckgen: [ 96%] 452440891 generated, 9676907 selectedtckgen: [ 96%] 452441071 generated, 9676913 selectedtckgen: [ 96%] 452441191 generated, 9676917 selectedtckgen: [ 96%] 452441371 generated, 9676920 selectedtckgen: [ 96%] 452441521 generated, 9676925 selectedtckgen: [ 96%] 452441681 generated, 9676931 selectedtckgen: [ 96%] 452441821 generated, 9676938 selectedtckgen: [ 96%] 452441951 generated, 9676940 selectedtckgen: [ 96%] 452442101 generated, 9676945 selectedtckgen: [ 96%] 452442281 generated, 9676948 selectedtckgen: [ 96%] 452442451 generated, 9676952 selectedtckgen: [ 96%] 452442641 generated, 9676958 selectedtckgen: [ 96%] 452442811 generated, 9676964 selectedtckgen: [ 96%] 452442991 generated, 9676968 selectedtckgen: [ 96%] 452443191 generated, 9676979 selectedtckgen: [ 96%] 452443371 generated, 9676983 selectedtckgen: [ 96%] 452443511 generated, 9676986 selectedtckgen: [ 96%] 452443671 generated, 9676991 selectedtckgen: [ 96%] 452443801 generated, 9676994 selectedtckgen: [ 96%] 452443951 generated, 9676999 selectedtckgen: [ 96%] 452444111 generated, 9677001 selectedtckgen: [ 96%] 452444291 generated, 9677004 selectedtckgen: [ 96%] 452444441 generated, 9677006 selectedtckgen: [ 96%] 452444641 generated, 9677007 selectedtckgen: [ 96%] 452444791 generated, 9677010 selectedtckgen: [ 96%] 452444961 generated, 9677015 selectedtckgen: [ 96%] 452445151 generated, 9677023 selectedtckgen: [ 96%] 452445311 generated, 9677026 selectedtckgen: [ 96%] 452445481 generated, 9677029 selectedtckgen: [ 96%] 452445651 generated, 9677031 selectedtckgen: [ 96%] 452445771 generated, 9677036 selectedtckgen: [ 96%] 452445901 generated, 9677040 selectedtckgen: [ 96%] 452446101 generated, 9677045 selectedtckgen: [ 96%] 452446231 generated, 9677047 selectedtckgen: [ 96%] 452446421 generated, 9677049 selectedtckgen: [ 96%] 452446551 generated, 9677052 selectedtckgen: [ 96%] 452446731 generated, 9677056 selectedtckgen: [ 96%] 452446891 generated, 9677058 selectedtckgen: [ 96%] 452447041 generated, 9677061 selectedtckgenHere goes my lengthy text.

I am not sure if one can avoid it.

An other inconvenient is that every thing is going to the error file (and not the log file). So when running a lot of command through a cluster one can not easily see if an error occur (since all error file are full of log)

for instance if you have a mrtrix command in a bash script (job.sh) doing
bash job.sh > log.log 2> error.log

all the output come to the error.log file. It is not a big deal but if you know a way to avoid it , to have only error in the error.log ?

thanks

Romain

2

Yes, I looked into this problem quite a bit a while back, have a look at this GitHub issue for the full details. I might take another look into it, it’s clearly going to affect quite a few users…

3

This is a bit trickier to do, since in MRtrix3 stdout is reserved for genuine command output (i.e. anything that might be used as input for another command, including in particular UNIX pipes), not user feedback. What you’re asking is essentially to use the -quiet option, which disables all non-warning/error level output, including the progressbar. Is that sufficient for your purposes?

One thing we could do is add a check for an environment variable to force quiet output. The advantage being that you’d only need to set for example export MRTRIX_QUIET=1 at the top of your script, and all MRtrix3 commands in that script would act as if -quiet had been included as an option.

Otherwise, to handle this right now, one simple option is use grep to inspect your logs. All error lines will contain the [ERROR] substring, so you should be able to run:

grep -F '[ERROR]' error.log

To restrict output to errors only. You can pipe to less if it’s a long list of errors:

grep -F '[ERROR]' error.log | less

Yet another option is to simply search for the keyword ERROR when viewing the log. Using less, you can start a search with / and type in your search string. So you’d run:

less error.log

And then type /ERROR (+Enter) to find the next error line. Hit n for the next match, N (shift+n) for the previous match. This is what I typically do…

4

One thing we could do is add a check for an environment variable to force quiet output. The advantage being that you’d only need to set for example export MRTRIX_QUIET=1 at the top of your script, and all MRtrix3 commands in that script would act as if -quiet had been included as an option.

Or you could just write your script making use of the MRtrix3 Python libraries, and this occurs automagically :stuck_out_tongue:

5

I’ve been looking into fixing the redirection issue. I’ve made a few changes that seem to work fine on my system, and should reduce the size of the logs drastically. It would be good to have users test it thoroughly on their HPC systems to make sure it actually does fix the problem - and doesn’t introduce others…

Instructions:

git fetch
git checkout fix_output_redirection
./build

(and to restore to master if needed: git checkout master && ./build)

The other issue you’d raised about ‘regular’ user feedback versus genuine errors going to separate logs is not something that I think we can ‘fix’. As I’d mentioned before, in MRtrix3 stdout is reserved for usable output - i.e. output that is intended to be used by subsequent commands, via pipelines or otherwise. This isn’t actually unique to MRtrix3 in any way, it’s a pretty standard Unix convention. The difference is that most Unix standard tools are designed for small quick tasks, and therefore don’t need to provide progress updates or other feedback. They invariably push their genuine output to stdout so it can be captured by scripts or passed through pipes, and any error messages or warnings go through stderr. If they do provide more feedback, then it has to be through stderr also, otherwise it would interfere with normal operation. One example is gzip -v, which provides feedback on the compression of each file as it is processed: this feedback goes out on stderr, since otherwise that might interfere with the output to stdout (when gzip is invoked with the -c option).

Clearly, this isn’t necessarily true for all commands - many never actually produce any user-visible output to stdout, in which case stdout is free to be used as the feedback channel (e.g. rsync). But for those commands that do expect to be used in pipelines, pushing all user feedback through to stderr really is the only sensible option…

There’s also another reason I’d argue that you really do want to keep your regular progress updates on the same stream as the errors and warnings. If you do encounter an error, all you would have on the error stream is these one or two lines, but crucially none of the context surrounding it. If your commands are allowed to proceed even when there has been a failure (the default in bash), then you might struggle to figure out at which point in the script the command failed, and not be able to look further up the log to see whether any of the preceding steps might explain the failure.

Anyway, enough for now. Let me know whether the changes I’ve made work out for you!

6

Hi Donald

Sorry for the late response… it is too late for me to test, but it seems to be incorporate it in the new release.
Great ! it works fine I have indeed much shorter log.

it is still in the stderr but ok. My point to put progress in the stdout, is that it make easier for HCP user to see if something goes wrong. when you run hunders of process
I just do a " cat err* " to see if I have any error line

but ok you have your points, and it make sense for specific use.

Many thanks

Romain

1 Like
7

I would be also interested in the implementation of an environment variable like MRTRIX_QUIET to force quiet output. My use case if for a long pipeline using both mrtrix and other software commands, where I am interested in the other command errors, which are currently lost in mrtrix outputs. Adding a -q option for every mrtrix command is kind of tedious. Thanks!

8

Just so you’re aware: @jdtournier is looking into this: https://github.com/MRtrix3/mrtrix3/pull/1182

9

Awesome, thanks!

1 Like
10

OK, that pull request is merged. You should be able to set MRTRIX_QUIET=1 (or any non-empty string) and get the -quiet behaviour that you’re after. Let me know if there are any issues.

As always:

git checkout master
git pull 
./build
11

Awesome, I will give it a try and let you know if I have issues.
Tim

1 Like