Setting a maximum demoninator for fractions

[Albert Chan]

(10-04-2023 01:03 AM)Albert Chan Wrote:  HP Prime HOME dot product work with higher precisions, thus more accurate.

Note: HOME DOT function extra precision is not coming from CAS side.

HOME> P:=PI                      → 3.14159265359
HOME> Q:=P*P                   → 9.86960440109
HOME> DOT([P,1],[P,-Q])     → 6.5ᴇ−13

P*P (exactly) = 9.8696044010906577398881
This suggested HOME DOT internally use 15 digits, truncated rounding.

CAS DOT is not the same as HOME DOT, even though they share the same name.

CAS> DOT([P,1],[P,-Q])       → 6.25277607469e−13

CAS float is 48 bits (truncated rounding), it is equivalent to IEEE double, stripped 53-48 = 5 bits.

lua> p, q = 3.14159265359, 9.86960440109
lua> p*p - q
6.590283874174929e-013

lua> hpfloat = fn'x: local hi,lo = bit.split(x); bit.join(hi,lo-lo%32)'
lua> p, q = hpfloat(p), hpfloat(q)
lua> hpfloat(p*p - q)
6.430411758628907e-013

lua> hpfloat(hpfloat(p*p) - q)
6.252776074688882e-013

It appeared CAS DOT doesn't use extended precision internally.