newRPL: [UPDATED April 27-2017] Firmware for testing available for download
|
10-22-2015, 09:22 PM
Post: #80
|
|||
|
|||
RE: newRPL: [UPDATED Oct-19-2015] Firmware for testing available for download
(10-22-2015 06:16 PM)Werner Wrote: But 527.67 x (5/9) does produce 293.15 ... I never said there's a rounding error, the number is computed properly. But 5/9 is not exact, and it's therefore marked as approximated. The "19.9:" was a normalization bug, the number was not normalized and confused the text conversion routine. When I show the number 0.55555555... I always add the ellipsis because there's a variable number of digits, up to 2000 digits but you are right, the last one is a 6. Anyway, the bug is fixed, and the calculation returns exactly 20. (or should I say exactly-approximatedly 20. :-) |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 2 Guest(s)