A quick precision test

[pito]

(06-04-2014 01:25 PM)Claudio L. Wrote:  [quote='Paul Dale' pid='13135' dateline='1401755816']
..
In human words: Set the precision to 34 digits to match the intermediate rounding you are using in WP34S. Then do the 27 squares by hand and the result is:

674530.4707410845593826891847277722 (if I didn't make a mistake copying by hand the numbers)

The "real" result at 34 digits is:
674530.4707410845593826891780297468

This is obtained from doing:

Code:


1.0000001 2 27 ^ ^

The difference is 6.698E-21, so there's 25 good digits out of the 34. Intermediate rounding only affected the last 9 digits.

Not sure whether doing 27 times x^2 or using Xpower2power27 is the same. Power functions are calculated in a different way usually (less precise I bet..).

I've calculated the stuff with my dusty vanilla decnum code (DECNUMDIGITS 34) and get:

Code:

// 27x x^2 TEST
cout<<"27xTEST"<<endl;
x = "1.0000001";
timer = millis;
for (i=1; i<=27; i++) {
  x = x * x;
}
timer = millis - timer;
cout<<"27xTest = "<<x<<endl;
cout<<"Elapsed x1 = "<<timer<<" ms"<<endl;

Code:

27xTEST
27xTest = 674530.4707410845593826891847277722
Elapsed x1 = 6 ms