A quick precision test

[Paul Dale]

(06-05-2014 11:03 AM)Claudio L. Wrote:  If I use 9 extra digits for internal calcs, then my result will only be affected by double rounding if my number is of the form:
nnnnn|499999999|5

There are other rounding cases. What about nnnnm|500000000|1 when m is odd and you are rounding to even? Or nnnnn|000000000|1 when you are rounding towards +infinity? Assuming you support rounding modes of course.

Agreed, the chance of being incorrect - if your internal result is correct - is low but it is still there. Correct rounding is a worthwhile, if somewhat quixotic, goal. One ULP is far more achievable.

Are you sure all of your guard digits are correct or even all the digits in your result? In general they (the guard digits) won't all be. There are many pitfalls for the unwary here. Even changing the order of operations can cause problems. You need to be amazingly careful to get all digits correct, if indeed this is even possible (which it isn't in general).

As an example (which isn't related to rounding), what does COS(1.57079632679489661923132169163975144) equal? Is it 2.09858469968755291048739331660132323e-36 or something different? I cheated here and downloaded your (rather nice BTW) newRPL demo and it gets the last thirteen digits incorrect. TAN of the same value loses fourteen digits and this answer is a huge number rather than something very tiny.

My original 34S test case was COS(1.570796326794896619231321691639751) where newRPL gets the last ten digits incorrect -- the 34S gets more digits incorrect here. With more memory available, I'd fix this.


- Pauli