Floating point bug in Canon F-789SGA

[Gerald H]

(11-30-2014 07:32 PM)Peter Van Roy Wrote:  I recently bought a Canon F-789SGA calculator because its specification intriguingly mentions that it has 18 digit precision. Tests show that it actually has 20 digit internal precision, and transcendental functions actually achieve this precision. That is the nice part.

Unfortunately, the calculator has bugs in its floating point operations. It took me a while to figure out what was going on, since there are several bugs that interact in unobvious ways. The main bug is an error in floating point addition. When doing a floating point addition A+B, if the difference in exponents between the two numbers A and B is 16, 17, 18, or 19, and if the lower four digits of the larger number are 0000, then the smaller number is not added in. That is, the lower four digits of the result stay 0000. For example, doing the addition 1 + 1E-16 gives exactly 1 instead of 1.0000000000000001. Floating point subtraction has a similar bug. There are other irregularities in the floating point operations, in particular, the value of the last (20th) digit also plays a role.

My guess to the origin of this bug is that the calculator is an improved version of a Casio calculator with 16 digit precision. Possibly the internal precision was increased without proper testing.

Has anyone else found similar bugs in these or other recent calculators?

i'm not sure it's a bug.

Handbook says:

"Precision +/- 1 at the 10th digit for a single calculation
+/- 1 at the least significant for exponential display" (page 10)

which I don't really understand, but could imply you are requiring accuracy above specifications.

Specs are certainly exceeded by the PFact function, where the greatest factor is declared "unfactored" whether prime or composite - in fact, the greatest factor found is prime for all values < 1009^2.

Overall a pleasant calculator & accurate & integration is annoyingly slow.