Round off error calculation example

[Dieter]

(07-23-2017 10:16 AM)Gamo Wrote:  I came across this example from youtube about the round off error on the older computer start from the earlier Pentium CPU which is about the floating point units. Here is the sample: ((2^54)+1)-(2^54) = 0 or 1 ?

I try this on two calculators Casio fx-300ES result is 0 and HP Prime result is 0
and try on the Google Search Calculator result is 0 and on the HP OmniGo 100 result is 0
Only from the WolfarmAlpha website gave result to 1

Sure. This is the expected result.

2^54 = 18 014 398 509 481 984. This is a number with 17 digits. The standard IEEE754 floating point format has 15-16 decimal digits. It works with 52 resp. 53 bit precision, that's why the example uses 2^54. Most calculators even work with merely 10 or 12 digits. So in any case 2^54 cannot be represented exactly, and of course adding or subtracting 1 does not make any difference as only the 17th digit is affected.

Example: on an 12-digit machine you get 2^54 = 1,80143985095 · 10^16. Add one and the result still is 1,80143985095 · 10^16. Finally subtract 1,80143985095 · 10^16 and – of course – you get zero.

BTW the Windows calculator works with 30+ digits, so it returns 1 here. ;-)

Dieter