Post Reply 
Numerical accuracy on the HP Prime (in Python, Home, and CAS)
07-26-2024, 04:27 PM
Post: #4
RE: Numerical accuracy on the HP Prime (in Python, Home, and CAS)
1e50 mod 2*pi can not be computed with any significant digit using "normal" precision. The reason is that the relative error rounding any floating point number will be say 1e-17 (it's 2^(-53) for double precision inside MicroPython, 2^(-48) for Xcas/CAS normal precision), that's an absolute error of 1e33 for 1e50, and that's much greater than 2*pi.
This is something that everyone doing scientific computation should know, if it's not teached in a course during highschool or first years of University, there is a real problem.
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: Numerical accuracy on the HP Prime (in Python, Home, and CAS) - parisse - 07-26-2024 04:27 PM



User(s) browsing this thread: 2 Guest(s)