Unexpected result calculating the determinant of a singular matrix (42S)

[Thomas Okken]

(10-22-2019 06:12 AM)Werner Wrote:  the 42S uses a*b-c*d when calculating the determinant of a 2x2 system, as you can see when you calculate the determinant of

1 2
3 1

and

1 2 0
3 1 0
0 0 1

The former returns -5 exactly, the latter -5.00000000001

So does that mean that the difference between the 42S and the 48G is that the former performs the calculation using only 12 digits? The number -5.00000000001 differs from the exact result in the 12th digit, while you'd expect to get the exact result if it performed the calculation in 15 digits and only rounded to 12 at the end.

The use of 15 digits would also explain how the 48G is able to get the exact result for Dave's example, regardless of the state of flag -54. That is, unless that's just a coincidence.