a b/c unexpected function

[Albert Chan]

(09-03-2020 06:21 PM)Joe Horn Wrote:  PDQ is the algorithm used by the second (and sometimes third) press. Its accuracy is controlled by the current display mode. For example, try Fixed 6 mode and see 0.000001 displayed as 1/500001 ...

What happens if mode is FIX 5 ?

For x = 1/10^6, PDQ'd fraction numerator cannot be more than 1

1/den - x < eps       → den > 1/(x + eps)

lua> x = 1/10^6
lua> function min_den(eps) return floor(1/(x + eps)) + 1 end
lua> min_den(1e-7), min_den(1e-6), min_den(1e-5)

909091       500001       90910

Although 1/90910 - x < 1e-5, we also have x - 0 < 1e-5

Does it meant PDQ'd 1e-6, FIX 5, should return the "simpler" 0/1 instead ?