Post Reply 
H->HMS conversion HP-15C vs. HP42S vs HP67
09-09-2018, 03:25 PM (This post was last modified: 09-14-2018 02:31 PM by Albert Chan.)
Post: #40
RE: H->HMS conversion HP-15C vs. HP42S vs HP67
(09-09-2018 01:54 PM)Thomas Okken Wrote:  The problem with using nextafter() is that the closest decimal equivalent isn't always going to be higher. In other words, you're always rounding up, but sometimes 0.3999 really is 0.3999, not a mangled 0.4000.

Please see my previous post #38. I decided not to do nextafter(), saving a step.
Always rounding-up also have a compensating always rounding-down.

Anyway, we are talking 1 ULP adjustment, not thousands (rounding to 12 digits).
The effect is so tiny (just enough to avoid the bug), you barely see it:

X = 0.3998999 ... (39 minutes, 99 seconds)
Y = X * 100.00000000000001 = 39.9900000000000019 ... ~ 39.99 (still 99 seconds)
DMS->DEG = 0.6774999999999999 (1.08 ULP below exact 39/60 + 99/3600)

X = 0.4699999 ... (47 minutes, but possible "40 seconds" bug)
Y = 47.000000000000007 ...
DMS->DEG = 0.7833333333333333 (0.07 ULP below exact 47/60)

Even if user input is 0.46999 99999 99999 8, above code still intepret it as 46 minutes 100 seconds.
So, no worries about mis-intepreting user input, 0.3999 will not mangled to 0.4000

Last 8 changed to 9 will confuse it (as 47 minutes). But, you are now reaching beyond IEEE double ...
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: H->HMS conversion HP-15C vs. HP42S vs HP67 - Albert Chan - 09-09-2018 03:25 PM



User(s) browsing this thread: 1 Guest(s)