Post Reply 
Bit's WP 34S and 31S patches and custom binaries (version: r3802 20150805-1)
05-26-2015, 07:05 AM
Post: #29
RE: Bit's WP 34S and 31S patches and custom binaries (version: r3747 20150207-1)
There are ways to verify the stack/volatiles clash.

Reduce the maximum matrix size and the inversion code will use less RAM. This would identify the issue but not help fix it (unless smaller matrices are okay 10x10 is pretty large).

Another option would be to rework the Doolittle LU decomposition to use decimal64's for the intermediate matrix rather than decimal128's. Again, this will show if memory is running out and provide a solution where the accuracy is reduced but the matrix operations still work up to 10x10.


- Pauli
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: Bit's WP 34S and 31S patches and custom binaries (version: r3747 20150207-1) - Paul Dale - 05-26-2015 07:05 AM



User(s) browsing this thread: 1 Guest(s)