Improving FIX mode

[Albert Chan]

(10-20-2023 08:07 AM)Pyjam Wrote:  I would expect value 0 to disable the switch to SCI completely. So, a true FIX mode.
0.00555 → FIX 2 → 0.01
0.000555 → FIX 2 → 0.00

I agree this is how FIX mode should work.
It is supposed to lose information! 0.00 is OK!

I do not understand how DIG setting work, and posted the question to issue.
Since DIG=0 match HP behavior, this is my guess.
For FIX-02, ULP = 0.01

DIG=0      HP behavior, flip to SCI if below 10^0/2 = 0.5 ULP      (*)
DIG=1      flip to SCI if below 10^1/2 = 5.0 ULP
...

For true FIX mode, DIG setting is negative, flip = 10^DIG/2 = 0.0

(*) Exception, no flip if FIX mode form already matched value.
     Example, any DIG setting, FIX mode of 0. = 0.000...