I’d not heard of the FSLvm, and I’m not surprised it won’t work with MRtrix3. I personally recommend you avoid that approach and use the WSL instead. Note that you’ll need two separate installs of MRtrix3 to get the full functionality: one within the WSL (along with FSL, ANTs, etc), and the other within MSYS2 for the graphics (since WSL doesn’t yet support that).
See this post for more details, and you’ll also find lots of discussion around that on this forum.