PROBLEM WITH PI π

[Albert Chan]

(12-29-2023 03:25 AM)bxparks Wrote:  I was shocked, because getting a "0" is impossible for a finite precision calculator (using base-2 or base-10)

Base 2, 1/3*3 - 1 = 0

float(1/3) = 1/3 ± ULP(1/3)/3      // sign depends on parity of bits
float(1/3)*3 = 1 ± ULP(1/3) = 1 ± ULP(1)/4 --> rounded back to 1

lua> 1/3*3 - 1
0

Base 10, we get non-zero result.

float(1/3) = 1/3 - ULP(1/3)/3
float(1/3)*3 = 1 - ULP(1/3) = 1 - ULP(1)/10 --> rounded below 1

HP71-B
>1/3*3 - 1
-.000000000001                        // = -1e-12 = -ULP(0.1)

TI is getting 0 because of its flush-to-zero "feature".