Yes, this issue can arise and has been discussed a few times before (e.g. here and here). We’ve thought about constraining the fit before, but never really took it further than that. If you have a use case where these things really matter, please let us know so we can figure out whether there’s anything we can or should do about it.
I note however that the command you mention (mrcalc -finite) wouldn’t fix FA<0 or FA>1 issues. It would only detect voxels that don’t contain not-a-number (NaN) or infinities, which is a different problem. These can be caused by negative values in the input DWI to dwi2tensor, but this shouldn’t happen with recent versions of MRtrix3 (since this commit in 2015). Please let me know if you still encounter NaNs or Infs in your outputs with recent versions of the software.