Dwifslpreproc issue python unhandled exception

Hello,

I am running into the following odd error for a couple of my subjects (but not all of them).

dwifslpreproc: [ERROR] Unhandled Python exception:
dwifslpreproc: [ERROR] TypeError: decode() takes no keyword arguments
dwifslpreproc: [ERROR] Traceback:
dwifslpreproc: [ERROR] /data/bswift-0/software/mrtrix3/bin/dwifslpreproc:182 (in execute())
dwifslpreproc: [ERROR] run.command('mrconvert ’ + path.from_user(app.ARGS.input) + ’ ’ + path.to_scratch(‘dwi.mif’) + grad_import_option + json_import_option + json_export_option)
dwifslpreproc: [ERROR] /a/ha-nfs-1-ib/export/data/bswift-0/software/mrtrix3/lib/mrtrix3/run.py:439 (in command())
dwifslpreproc: [ERROR] stdout_text = finalise_temp_file(process.iostreams[0])
dwifslpreproc: [ERROR] /a/ha-nfs-1-ib/export/data/bswift-0/software/mrtrix3/lib/mrtrix3/run.py:432 (in finalise_temp_file())
dwifslpreproc: [ERROR] contents = stream.read().decode(‘utf-8’, errors=‘replace’)
dwi_prep.mif: Command not found.

After deleting, retrying, and starting from scratch the script ran without error.

Sorry to reopen this but unfortunately I am still getting the same error now with additional subjects and deleting and rerunning does not appear to solve the issue.

OK, the issue boils down to this:

dwifslpreproc: [ERROR] TypeError: decode() takes no keyword arguments

If I search for that specific error online, the only hits I get relate to Python 2.6, which we explicitly do not support (it’s far too old…). Can you double-check which version of python you have running, e.g. using:

python --version

If it’s anything older than 2.7, then that’s the reason for the failure - you’ll need to use a more up to date version, preferably python >3.6 (I think). Note that we will stop supporting versions of python older than 3.6 in the next release…

login-1: python --version
Python 3.7.3

OK, that’s really weird. That version should be fine, but your error message is definitely suggestive of an older python version. Can we try setting up a small test python script to see what it reports?

Try this copy/pasting this as-is exactly into the terminal, and press enter:

cat > test_py <<EOD
#!/usr/bin/env python
import sys
print (sys.version_info)
EOD
chmod a+x test_py
./test_py

On my system, this returns:

sys.version_info(major=3, minor=10, micro=4, releaselevel='final', serial=0)

I’d like to see what this returns on yours – it replicates the shebang line we use in our scripts.

Hmm. I got the following.

login-1:/data/bswift-1/dcallow/Diffusion: cat > test_py <<EOD
? #!/usr/bin/env python
? import sys
? print (sys.version_info)
? EOD
login-1:/data/bswift-1/dcallow/Diffusion: chmod a+x test_py
login-1:/data/bswift-1/dcallow/Diffusion: ./test_py
(2, 4, 1, ‘final’, 0)

Sorry, the system I am running it on actually has the following output.

sys.version_info(major=3, minor=7, micro=3, releaselevel=‘final’, serial=0)

And is it likely due to the top error or could the route issue be due to this lower error line.

dwifslpreproc: [ERROR] contents = stream.read().decode(‘utf-8’, errors=‘replace’)

Not sure what to make of this…

That’s indeed the line that causes the exception to be raised, yes. It’s saying that the .decode() method doesn’t expect named arguments. And as you can see from the official Python docs, support for named arguments was added to this method in version 2.7… Which is why I’m struggling to understand what might be going on. Things are a bit different in version 3.7, in that we’re relying on the decode() method from the bytes class rather than the str class, but it should also work fine – though I note that support for keyword arguments was added to that method in version 3.1. But since both relevant versions you quote are 3.7.3 or 3.10.4, that doesn’t apply either.

So I still think the evidence suggests that an old version of python is being used for some reason, though I’m not sure how or why.

To totally rule this out, would you mind adding this line near the end of the dwifslpreproc file (which you’ll find in the bin/ folder of your MRtrix install), trying your command again as you did when it last failed, and double-checking which version is being used in that context?

# Execute the script
import mrtrix3
print (sys.version_info)   # <= add this line
mrtrix3.execute() #pylint: disable=no-member        

Unfortunately, I am running this on a linux system where I don’t have permission to add this to the software.

Still running into the following error.

dwifslpreproc:
dwifslpreproc: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-hel$dwifslpreproc:
dwifslpreproc: Generated scratch directory: /a/ha-nfs-2-ib/export/data/bswift-1/dcallow/Diffusion/sub-ams1019/ses-ex/dwifslpreproc-tmp-Y7HNIK/
Command:  mrconvert /a/ha-nfs-2-ib/export/data/bswift-1/dcallow/Diffusion/sub-ams1019/ses-ex/dwi_denoised_unringed.mif /a/ha-nfs-2-ib/export/data/bswift-1/dcallow/Diffusion/sub-ams1019/ses-ex$
dwifslpreproc: [ERROR] Unhandled Python exception:
dwifslpreproc: [ERROR]   TypeError: decode() takes no keyword arguments
dwifslpreproc: [ERROR] Traceback:
dwifslpreproc: [ERROR]   /data/bswift-0/software/mrtrix3/bin/dwifslpreproc:182 (in execute())
dwifslpreproc: [ERROR]     run.command('mrconvert ' + path.from_user(app.ARGS.input) + ' ' + path.to_scratch('dwi.mif') + grad_import_option + json_import_option + json_export_option)
dwifslpreproc: [ERROR]   /a/ha-nfs-1-ib/export/data/bswift-0/software/mrtrix3/lib/mrtrix3/run.py:439 (in command())
dwifslpreproc: [ERROR]     stdout_text = finalise_temp_file(process.iostreams[0])
dwifslpreproc: [ERROR]   /a/ha-nfs-1-ib/export/data/bswift-0/software/mrtrix3/lib/mrtrix3/run.py:432 (in finalise_temp_file())
dwifslpreproc: [ERROR]     contents = stream.read().decode('utf-8', errors='replace')

Also note that I get this error whether I use the rpe_header or rpe_all option

I am unsure why but when I changed the following in my script.

#source /data/bswift-0/software/gcc-6.3.0/load_gcc_6.3.0.sh
#source /data/bswift-0/software/loadpython.csh 3.7

and just did module load gcc/6.1.0 and module load python/3.7.3 before running the script and it seems to have no issue running anymore.

1 Like

Hmm. After the above fix I actually am getting the following error now.

dwifslpreproc: Changing to scratch directory (/a/ha-nfs-2-ib/export/data/bswift-1/dcallow/Diffusion/sub-ams1019/ses-ex/dwifslpreproc-tmp-8KI4UG/)
Command:  dirstat dwi.mif -output asym
Command:  mrconvert dwi.mif - | dwiextract - se_epi.mif -bzero
Command:  mrinfo dwi.mif -export_grad_mrtrix grad.b
Command:  mrconvert se_epi.mif topup_in.nii -strides -1,+2,+3,+4 -export_pe_table topup_datain.txt
Command:  topup --imain=topup_in.nii --datain=topup_datain.txt --out=field --fout=field_map.nii.gz --config=/data/bswift-0/software/fsl6.0.1/etc/flirtsch/b02b0.cnf --verbose
Traceback (most recent call last):
  File "/a/ha-nfs-1-ib/export/data/bswift-0/software/mrtrix3/lib/mrtrix3/app.py", line 197, in _execute
    module.execute()
  File "/data/bswift-0/software/mrtrix3/bin/dwifslpreproc", line 795, in execute
    topup_output = run.command(topup_cmd + ' --imain=topup_in.nii --datain=topup_datain.txt --out=field --fout=field_map' + fsl_suffix + ' --config=' + topup_config_path + ' --verbose' + topu$  File "/a/ha-nfs-1-ib/export/data/bswift-0/software/mrtrix3/lib/mrtrix3/run.py", line 453, in command
    raise MRtrixCmdError(cmdstring, return_code, return_stdout, return_stderr)
mrtrix3.run.MRtrixCmdError: SSD = 695.074       n = 113832      Reg = 0 Cost = 695.074
SSD = 505.101   n = 113832      Reg = 26.2798   Cost = 531.38
SSD = 434.812   n = 113817      Reg = 52.0799   Cost = 486.892
SSD = 415.845   n = 113647      Reg = 63.8523   Cost = 479.697
SSD = 410.453   n = 113491      Reg = 67.7025   Cost = 478.156
SSD = 408.804   n = 113371      Reg = 68.8802   Cost = 477.684
***Going to next resolution level***

Asked the person in charge of the linux system to add the following to the script and setting up to rerun to get an idea of the system error.

I got the following…

dwifslpreproc -force dwi_denoised_unringed.mif dwi_prep.mif -rpe_header
sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)
dwifslpreproc:
dwifslpreproc: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-help option) for more information.
dwifslpreproc:
mrinfo: [ERROR] failed to open key/value file "/a/ha-nfs-2-ib/export/data/bswift-1/dcallow/Diffusion/dwi_denoised_unringed.mif": No such file or directory
mrinfo: [ERROR] error opening image "/a/ha-nfs-2-ib/export/data/bswift-1/dcallow/Diffusion/dwi_denoised_unringed.mif"

print (sys.version_info)

Any ideas on what could be causing the error?