Rounding errors when mimicking Mod() with FP()
|
03-14-2016, 08:14 AM
Post: #4
|
|||
|
|||
RE: Rounding errors when mimicking Mod() with FP()
Hello
Similar rounding errors are inevitable when working with floating point numbers. Various floating point (FP) systems will behave differently depending on their implementation (binary/decimal, number of internal/external digits used for precision). Some system will try to hide the "problems" (which itself causes other issues), other will not. Some will hide it by hiding the real value of numbers (which means that subsequent calculations are technically OK, but yield a 'strange' result), some by modifying the numbers... Fixing a given system for a given case usually breaks it for another case :-) Here, you want to do exact arithmetics. This can be done using binary integers or the CAS. use #10d^R->B(X) MOD #13d for example for F1(X) and you get the result that you want (warning, the function is only defined for integer X.). You can go to the num setup, set step to 1 and got to NUM view to see the data. irem(10^X,13) should also do what you want (this is a CAS function, integer remener). Cyrille Although I work for the HP calculator group, the views and opinions I post here are my own. I do not speak for HP. |
|||
« Next Oldest | Next Newest »
|
Messages In This Thread |
Rounding errors when mimicking Mod() with FP() - StephanP - 03-13-2016, 03:56 PM
RE: Rounding errors when mimicking Mod() with FP() - Fortin - 03-13-2016, 06:34 PM
RE: Rounding errors when mimicking Mod() with FP() - StephanP - 03-14-2016, 07:30 AM
RE: Rounding errors when mimicking Mod() with FP() - Fortin - 03-14-2016, 11:18 AM
RE: Rounding errors when mimicking Mod() with FP() - cyrille de brébisson - 03-14-2016 08:14 AM
|
User(s) browsing this thread: 1 Guest(s)