WP34s: Domain Error in RMDR and MOD
|
10-21-2015, 09:39 PM
Post: #2
|
|||
|
|||
RE: WP34s: Domain Error in RMDR and MOD
The decNumber mod function isn't efficient and requires enough digits to represent the source numbers and the result (i.e. 2*384 + 16). I increase the number of carried digits for these operations to a little above this so that single precision never fails. I didn't go as far as supporting double precision -- there isn't sufficient RAM to do this directly.
Making double precision inputs work across the entire range will require a different algorithm and since we don't guarantee any results in double precision I didn't bother - Pauli |
|||
« Next Oldest | Next Newest »
|
Messages In This Thread |
WP34s: Domain Error in RMDR and MOD - Dieter - 10-21-2015, 12:13 PM
RE: WP34s: Domain Error in RMDR and MOD - Paul Dale - 10-21-2015 09:39 PM
RE: WP34s: Domain Error in RMDR and MOD - Dieter - 10-22-2015, 01:00 PM
RE: WP34s: Domain Error in RMDR and MOD - Paul Dale - 10-22-2015, 09:57 PM
|
User(s) browsing this thread: 1 Guest(s)