Post Reply 
FORTRAN floating point accuracy problems
04-13-2016, 06:23 AM
Post: #43
RE: FORTRAN floating point accuracy problems
1E15/(5E15 +1) on Free42 Bin equals (hex digitsSmile

1.9999999999998 * 2^(-3)

In the display of course, this shows as 0.2, but after

1e8 x FP

you see

9.99999996275e-1

And the '6' in its decimal representation is really the 16th digit, not the 17th (the leading '1' is the implied bit, not a full digit), it doesn't get rounded.

When you enter 0.2, it's hex representation is

1.999999999999A * 2^(-3)

Cheers, Werner

41CV†,42S,48GX,49G,DM42,DM41X,17BII,15CE,DM15L,12C,16CE
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: FORTRAN floating point accuracy problems - Werner - 04-13-2016 06:23 AM



User(s) browsing this thread: 6 Guest(s)