Post Reply 
newRPL: [UPDATED April 27-2017] Firmware for testing available for download
10-22-2015, 05:56 PM
Post: #76
RE: newRPL: [UPDATED Oct-19-2015] Firmware for testing available for download
(10-22-2015 02:12 PM)Werner Wrote:  
(10-11-2015 01:27 AM)Claudio L. Wrote:  The third case is a bug in the displaying routine. It shows bad rounding of the last digit, from a 9 into a ':' which is the ASCII character that follows 9, when it should actually propagate the carry to the next digit. I'll work on that.

As usual, thanks for the bug reports.

Claudio, 68 F° is exactly 20°C, so there should be no rounding?
Cheers, Werner

But there is, because the order of operations when the unit conversion takes place is not altered in any way, operations are done as they come and that can cause inexact operations to take place.

68_°F is first converted to absolute scale by adding 459.67. Then the unit is scanned for replacement, and 1_°R = 5/9_K, so there's 527.67*(5/9), and there's where the rounding happens: 5/9 is a real and cannot be represented exactly, so it's .555555555..... and that multiplication produces 293.149999999999999999999....
Last, the scale is converted back to Celsius by subtracting 273.15, resulting in 19.999999999999999999999....
After I fixed a normalization bug, it now gets correctly rounded to 20., but I can't get rid of the dot (approximated), since the number is not exact. It's 20 up to the system precision (not just displayed as 20, it's internal representation is exactly 20), but because it is the result of operating with an approximated number, the system marks it with the trailing dot.
Find all posts by this user
Quote this message in a reply
Post Reply 

Messages In This Thread
RE: newRPL: [UPDATED Oct-19-2015] Firmware for testing available for download - Claudio L. - 10-22-2015 05:56 PM

User(s) browsing this thread: