Improving FIX mode

[c3d]

(10-20-2023 10:59 AM)Albert Chan Wrote:  

(10-20-2023 08:07 AM)Pyjam Wrote:  I would expect value 0 to disable the switch to SCI completely. So, a true FIX mode.
0.00555 → FIX 2 → 0.01
0.000555 → FIX 2 → 0.00

I agree this is how FIX mode should work.
It is supposed to lose information! 0.00 is OK!

It depends on your goal. I think this is an acceptable and even useful output, but it definitely does not match what any HP calculator does that I know of. At the moment, I'm first trying to be close to what people are used to, and second only trying to offer useful improvements.

Quote:I do not understand how DIG setting work, and posted the question to issue.
Since DIG=0 match HP behavior, this is my guess.
For FIX-02, ULP = 0.01

DIG=0      HP behavior, flip to SCI if below 10^0/2 = 0.5 ULP      (*)
DIG=1      flip to SCI if below 10^1/2 = 5.0 ULP
...

Interestingly, I just noticed that the HP Prime corresponds to the DIG 1 setting, not DIG 0. It does not display 0.01 but 5.50E-3. In other words, there is not one HP behavior but at least two, which DB48X now both supports.

Quote:For true FIX mode, DIG setting is negative, flip = 10^DIG/2 = 0.0

Today, DIG setting cannot be negative. But getting what you call "true" FIX mode does seem somewhat consistent with the idea of a negative DIG value.

I would not call it a "true" FIX mode in the sense that to me, "true" means "HP-like" in that instance.

Quote:(*) Exception, no flip if FIX mode form already matched value.
     Example, any DIG setting, FIX mode of 0. = 0.000...