Programming the INT and FRAC functions
|
01-05-2016, 07:37 PM
Post: #16
|
|||
|
|||
RE: Programming the INT and FRAC functions
(01-05-2016 12:42 PM)Pekis Wrote: While the number 15.9999999999 (kept in internal memory) is displayed as 16, should the INT function have to display 15 (applying INT directly) or 16 (Rounding to 10 digits then applying INT) ? Of course any function, including INT, should work on the exact value and not on the one rounded to display precision. Just imagine what would happen in FIX 0 mode. So the result of course has to be 15. If you want to do it the other way round, simply have the number explicitely rounded before the function is called. It's the programmer's job to take care of results that may come out as 15,9999999999 although the expected value is 16. This applies to any function. Even a simple x=y? test may go wrong. Luckily most calculators use BCD arithimetics – in binary things may be much worse. Dieter |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 1 Guest(s)