Wow, that’s some impressive detective work! 
It looks like a bug in FiberNavigator, unfortunately… There’s many ways to represent a NaN, according to the IEEE floating-point standard:
A NaN (Not a Number) can be represented with any of the many bit patterns that satisfy the definition of a NaN. The hex value of the NaNshown in TABLE 2-3 is just one of the many bit patterns that can be used to represent a NaN.
MatLab obviously uses one particular (and valid) representation, while in C++, we end up with another, equally valid representation. There are functions to check for NaNs that get will get it right (and this is what we use in our code, which is why it accepts these data). So my guess is that FIberNavigator is relying on a hard-coded value and checking for a bitwise match, rather than using the recommended isnan() function. I reckon it might be worth alerting them to this issue.
In the meantime, I think you might be able to modify the MatLab code to give it your own version of a NaN. First off, you’ll need to define your new NaN:
mynan=uint32(hex2dec('7fc00000'));
and then write those instead, by replacing this line with:
fwrite (f, [ mynan mynan mynan ], 'uint32');
That might get around the issue for you… 