H->HMS conversion HP-15C vs. HP42S vs HP67
|
09-07-2018, 02:40 PM
Post: #22
|
|||
|
|||
RE: H->HMS conversion HP-15C vs. HP42S vs HP67
(09-07-2018 09:43 AM)Werner Wrote:(09-06-2018 05:31 PM)Dieter Wrote: it looks like the usual binary roundoff problem. The approach I chose for ->HR for the binary version is robust but not very pretty: multiply x by 1e12, round to nearest integer, and then do everything using integer math. Plus, there is a special case in for |x| < 0.01, where the conversion reduces to a simple division by 0.36. The idea was to make sure that it will never get anything wrong that the HP-42S gets right, but the simplistic approach I chose does mean a bit of precision is lost. If someone has a better algorithm lying around for IEEE-754 double precision binary, I would be interested. |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 11 Guest(s)