Post Reply 
H->HMS conversion HP-15C vs. HP42S vs HP67
09-08-2018, 03:45 PM
Post: #29
RE: H->HMS conversion HP-15C vs. HP42S vs HP67
(09-08-2018 02:35 PM)Thomas Okken Wrote:  It doesn't multiply the entire number by 10^12, just the fractional part. But, yes, with a case like that, numerical cancellation will cause the binary round-off to mess up the result. Avoiding that is possible but ugly (I think the 33S does this).

I thought of that (scale only fractional part) before posting.
But, whether or not the integer part got removed before scaling does not matter:

X - INT(X) is exact = 0.03999996185 ... (DMS->DEG over-counted 40 seconds still here)

In fact, my suggestion of rounded 15 digits required INT(X) not to be removed.

Rounded X (to 15 digits) = "1000000000.04" (character string)
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: H->HMS conversion HP-15C vs. HP42S vs HP67 - Albert Chan - 09-08-2018 03:45 PM



User(s) browsing this thread: 19 Guest(s)