New Acron RPN Calculator for Android

[vanLudwig]

(06-20-2016 01:43 PM)rprosperi Wrote:  I have not had a chance to explore precision levels yet, but some odd things happen, e.g. entering 10 E-324[ENTER] yields 9.88131281682493 E-324.

You are running into the limitations of floating point numbers. For 64-bit floating point values, the smallest positive number (referred to as machine epsilon) is 4.94065645841247E-324. The second smallest number would be 2 * epsilon (which is what you were seeing), the third smallest number would be 3 * epsilon. This continues for the first 2^51 numbers; then the binary exponent increases and so does the gap between representable numbers.

(06-20-2016 01:43 PM)rprosperi Wrote:  Also, 2 [ENTER] 3 [/] => 6.6666666664833 E-7.

Can you explain this further? It seems like you were pointing out a rounding error, but 2/3 is 0.6666667, not 6.66667E-7. Either way, I can't replicate the round off error or the decimal being wrong by 7 places.