Unexpected result calculating the determinant of a singular matrix (42S)

10222019, 11:11 AM
(This post was last modified: 10222019 11:20 AM by Thomas Okken.)
Post: #22




RE: Unexpected result calculating the determinant of a singular matrix (42S)
(10222019 06:12 AM)Werner Wrote: the 42S uses a*bc*d when calculating the determinant of a 2x2 system, as you can see when you calculate the determinant of So does that mean that the difference between the 42S and the 48G is that the former performs the calculation using only 12 digits? The number 5.00000000001 differs from the exact result in the 12th digit, while you'd expect to get the exact result if it performed the calculation in 15 digits and only rounded to 12 at the end. The use of 15 digits would also explain how the 48G is able to get the exact result for Dave's example, regardless of the state of flag 54. That is, unless that's just a coincidence. 

« Next Oldest  Next Newest »

User(s) browsing this thread: