if there aren't too many, stick FWAITS in there and take them out until the problem pops up again
I would do it a bit differently if you know which parameters always generate that NAN (and the time it takes to arrive at those parameters is relatively short).
a) Put a break point after the last FPU instruction. I would also change that instruction to exit the program; the first run will be to make sure that the NAN gets produced.
b) Leave that first break point and insert a second break point behind the preceeding FPU instruction, and restart the program; check after that breakpoint if the NAN got produced.
c) Remove that second breakpoint and continue to move it behind the preceeding FPU instruction until the NAN stops being produced. The next FPU instruction would then be the main culprit. Check the content of all the FPU/CPU registers along with the expected effect of the next FPU/CPU instructions on such content. You may also check if the NAN still got produced at the end of the computation; it could be an important clue.
Have fun.
Edit:
If you have several possible branches in your algo, you may also need to check those other branches if the program doesn't stop at the "second" break point.
