"Why do calculators get this wrong?" (YouTube)

[EdS2]

(07-19-2020 05:38 PM)Thomas Okken Wrote:  When I calculate 11^6/13/pi*3600, I get 156158412.9999625068535714091394323 which is rather close to being an integer...

Good thought: perhaps the Casio divides by pi and then multiplies successively by the small prime factors of 25200 until it sees something rather close to an integer.

And yet, the failing example with 17 instead of 13 also comes out very close to an integer:
119415256.9999713...
In fact, even closer. Very odd.

Let's just see what happens if we do each step at 12 digits, rounded:
11^6/13 =[12r] 136273.923077
/pi[12r] =[12r] 43377.3369445
*3600 =[12r] 156158413 (integer)

whereas
11^6/17 =[12r] 104209.470588
/pi[12r] =[12r] 33170.9047221
*3600 =[12r] 119415257

But suppose we calculate to 12 digits and truncate instead of rounding:
11^6/13 =[12t] 136273.923076
/pi[12t] =[12t] 43377.3369442
*3600 =[12t] 156158412.999

11^6/17 =[12t] 104209.470588
/pi[12t] =[12t] 33170.9047222
*3600 =[12t] 119415256.999

Let's try again but with a rounded value of pi:
11^6/13 =[12t] 136273.923076
/pi[12r] =[12t] 43377.3369441
*3600 =[12t] 156158412.998

11^6/17 =[12t] 104209.470588
/pi[12r] =[12t] 33170.9047221
*3600 =[12t] 119415256.999

Not helping! I should perhaps have done 12 digit arithmetic throughout, instead of rounding or truncating a more precise number.

Do I have a 12 digit calculator... yes, I have an HP-30.
So, the 13 calculation comes to an integer.
As does the 17...
So I'm no closer!