Have you inspected the normalised DWI data? I’d suspect you see the same scaling issue for all shells which points at normalisation gone wrong. If you used dwibiascorrect and have multi-shell data, try mtnormalise instead or adjust the N4 options (see for instance here and here.). If you used mtnormalise, make sure the brain mask is very close to or within brain parenchyma.