As far as I can tell you are running WSL on Windows 10. Is that correct?
Note that, currently, this is not the recommended way to install MRtrix3 on Windows.
The recommended approaches use MSYS2 and are described here:
- https://www.mrtrix.org/download/windows-msys2/ if you want to use a binary installer
- https://mrtrix.readthedocs.io/en/3.0.2/installation/build_from_source.html if you want to install from source.
The error you are seeing is a known issue with QT on WSL that affects running the graphical user interface of MRtrix and is described here: https://github.com/MRtrix3/mrtrix3/issues/2048.
If you really want to proceed with installing MRtix3 in WSL, @jdtournier created a wiki entry dedicated specifically to this at Installing on Windows Subsystem for Linux (WSL).