FORTRAN floating point accuracy problems

[Werner]

(04-07-2016 02:28 PM)Dieter Wrote:  

(04-07-2016 01:36 PM)Werner Wrote:  BTW with x = 1e13/(5e13 + 1), on Free42 Binary, INT(100*x) delivers 19 and not 20 as you stated,

That's an easy case that is handled correctly even by Excel. Replace the "E13" part by E14 or E15, depending on Free42's working precision, so that the "+1" occurs in the least significant digit.

Dieter

But it's your example a few posts up :

Quote:Consider the case dms = 1/5,000 00000 00001

of which you claimed Excel's INT function of 100*dms yielded 20:

Quote:Now INT of this yields 20 exactly (not 19!)

Well, never mind. If you do want to use d.ms format, which can't be represented exactly in binary floating point, you'll need rounding.

Cheers, Werner