Post Reply 
WP34s: Domain Error in RMDR and MOD
10-21-2015, 09:39 PM
Post: #2
RE: WP34s: Domain Error in RMDR and MOD
The decNumber mod function isn't efficient and requires enough digits to represent the source numbers and the result (i.e. 2*384 + 16). I increase the number of carried digits for these operations to a little above this so that single precision never fails. I didn't go as far as supporting double precision -- there isn't sufficient RAM to do this directly.

Making double precision inputs work across the entire range will require a different algorithm and since we don't guarantee any results in double precision I didn't bother Smile


- Pauli
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: WP34s: Domain Error in RMDR and MOD - Paul Dale - 10-21-2015 09:39 PM



User(s) browsing this thread: 1 Guest(s)