H->HMS conversion HP-15C vs. HP42S vs HP67
|
09-08-2018, 03:45 PM
Post: #29
|
|||
|
|||
RE: H->HMS conversion HP-15C vs. HP42S vs HP67
(09-08-2018 02:35 PM)Thomas Okken Wrote: It doesn't multiply the entire number by 10^12, just the fractional part. But, yes, with a case like that, numerical cancellation will cause the binary round-off to mess up the result. Avoiding that is possible but ugly (I think the 33S does this). I thought of that (scale only fractional part) before posting. But, whether or not the integer part got removed before scaling does not matter: X - INT(X) is exact = 0.03999996185 ... (DMS->DEG over-counted 40 seconds still here) In fact, my suggestion of rounded 15 digits required INT(X) not to be removed. Rounded X (to 15 digits) = "1000000000.04" (character string) |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 19 Guest(s)