Post Reply 
H->HMS conversion HP-15C vs. HP42S vs HP67
09-08-2018, 06:03 PM (This post was last modified: 09-08-2018 06:08 PM by Albert Chan.)
Post: #30
RE: H->HMS conversion HP-15C vs. HP42S vs HP67
(09-07-2018 03:43 PM)Albert Chan Wrote:  Example: if angle = 2° 1' , X = 2.01 (IEEE double of 2.0099999...)

X = 2.0099999 ... --> Y = 20100.00 (scaled and rounded to hundredth of second)

DMS->DEG: (Y - 40*int(Y/100) - 2400*int(Y/10000)) / 3600 = (20100 - 8040 - 4800) / 3600 ~ 2.0166666...

Uh, I was getting stupid ...
My post #23 already have an elegant method.

Scale by 10000 (to move the troublesome decimal point), then round to 15 digits.
(approximated rounding good enough, just to catch the "40 seconds" bug)

That is all that is needed. Smile
No reason to get precisely rounded 15 digits decimal.
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: H->HMS conversion HP-15C vs. HP42S vs HP67 - Albert Chan - 09-08-2018 06:03 PM



User(s) browsing this thread: 16 Guest(s)