Unexpected result calculating the determinant of a singular matrix (42S)

[ttw]

(10-21-2019 11:58 AM)John Keith Wrote:  The HP 50 in approximate mode and with flag -54 (the "cheat" flag) set*, returns 0 for both Dave's and Thomas's matrices. I'm surprised that the 42s gives an inexact result, I assumed the internal code for matrix math was essentially the same as the 48/49 series.


* Note that setting the flag removes the "cheat". Also, according to the HP 50 AUR, the basis for setting tiny elements to 0 is that intermediate results are less than 1E-14. It does not say anything about integer values.

Approximately true. However, I've been working on some number theory stuff (which I'll post if I get it in shape for public use) that use really big (greater than 2^64) integers. Some seemingly integer stuff ends up being floating point. Conversions of big numbers out of binary comes to mine. I've got most things to work.

Reducing some things with big integer multiple of things like Sqrt(2) or (Sqrt(5)-1)/2 and the like need careful handling. Generally FLOOR and CEIL work well. FXND can be a problem as I found some case (I can't reproduce it) where would convert a number to a floating point.

That's for theory. In practice, I can just keep the numerators and denominators of stuff separate and use really close rational approximations for the irrational numbers in the final step.