Post Reply 
"Why do calculators get this wrong?" (YouTube)
07-22-2020, 08:11 AM
Post: #13
RE: "Why do calculators get this wrong?" (YouTube)
(07-20-2020 02:37 PM)EdS2 Wrote:  Thanks Albert - 13 digits might well be the solution to the problem.
I should have said more: it's the idea that multiplying by 3600 and testing for an integer result is a likely approach, combined with the observation that the two cases produce results with different near-integer sizes, such that one case has more digits beyond the point. So the closer match of the /17 case no longer looks closer, because the /13 case lost one digit of precision.

Very interesting.
Find all posts by this user
Quote this message in a reply
Post Reply 

Messages In This Thread
RE: "Why do calculators get this wrong?" (YouTube) - EdS2 - 07-22-2020 08:11 AM

User(s) browsing this thread: 1 Guest(s)