H->HMS conversion HP-15C vs. HP42S vs HP67
|
09-09-2018, 06:13 AM
(This post was last modified: 09-13-2018 02:43 PM by Albert Chan.)
Post: #34
|
|||
|
|||
RE: H->HMS conversion HP-15C vs. HP42S vs HP67
Hi, Joe Horn
I was in the middle of removing Pekins formula ... DMS->DEG may have problems too, with INT(100*X) Doing conversion with integer degree and minutes is safer (float seconds OK): Y = 100*X Y = nextafter(Y, Y + Y) DMS->DEG = (2.5*Y - int(Y) - 60*int(Y/100)) / 90 Max error = decimal to binary error + scaling error = 0.5 + 0.5 = 1.0 ULP --> 1 ULP is enough to get to safety, away from "40 seconds bug" Edit: conclusion is correct, but ULP math was wrong, see post 55 |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 13 Guest(s)