Post Reply 
H->HMS conversion HP-15C vs. HP42S vs HP67
09-09-2018, 06:13 AM (This post was last modified: 09-13-2018 02:43 PM by Albert Chan.)
Post: #34
RE: H->HMS conversion HP-15C vs. HP42S vs HP67
Hi, Joe Horn

I was in the middle of removing Pekins formula ...
DMS->DEG may have problems too, with INT(100*X)
Doing conversion with integer degree and minutes is safer (float seconds OK):

Y = 100*X
Y = nextafter(Y, Y + Y)
DMS->DEG = (2.5*Y - int(Y) - 60*int(Y/100)) / 90

Max error = decimal to binary error + scaling error = 0.5 + 0.5 = 1.0 ULP
--> 1 ULP is enough to get to safety, away from "40 seconds bug"

Edit: conclusion is correct, but ULP math was wrong, see post 55
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: H->HMS conversion HP-15C vs. HP42S vs HP67 - Albert Chan - 09-09-2018 06:13 AM



User(s) browsing this thread: 13 Guest(s)