From your error message I take that you did not use homebrew but tried to compile mrtrix3 yourself. You can install the dependencies with homebrew as described here:
Install Qt5: brew install qt5
Install pkg-config: brew install pkg-config
Add Qt’s binaries to your path: export PATH=`brew --prefix`/opt/qt5/bin:$PATH
and then repeat ./configure && ./build
(in the same shell).