New firmware for the Swissmicros DM15L
|
10-22-2024, 12:39 PM
(This post was last modified: 10-22-2024 12:47 PM by Johnp_g.)
Post: #126
|
|||
|
|||
RE: New firmware for the Swissmicros DM15L
I found another weird rounding error!
in FIX04 1.00005 ENTER Displays as 2.0000 change to FIX05 now shows 1.00005 In FIX03 displays as 1.000 Staying in FIX03 1.0005 ENTER displays as 2.000 It looks as though if there's a digit in the "n+1" position, in FIX0n (or SCI0n) that the rounding up happens to the whole number, not to the decimal portion. This seems to be the case whatever n is chosen for FIX/SCI/ENG. I found it while going through the HP15C User Manual examples. on P.22 "Numeric Functions" in FIX04 3.4012 e^x displays as 31.0000 instead of 30.0001 weirder still..... with 31.0000 in the display pressing ENTER puts 31.0000 in Y and X add them "+" display now shows 60.0002 so the underlying number was still actually 30.0001 but was displayed as 31.0000, even when ENTER duplicated it and raised it to Y register! very confusing! Cheers John |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 8 Guest(s)