FORTRAN floating point accuracy problems
|
04-13-2016, 06:23 AM
Post: #43
|
|||
|
|||
RE: FORTRAN floating point accuracy problems
1E15/(5E15 +1) on Free42 Bin equals (hex digits
1.9999999999998 * 2^(-3) In the display of course, this shows as 0.2, but after 1e8 x FP you see 9.99999996275e-1 And the '6' in its decimal representation is really the 16th digit, not the 17th (the leading '1' is the implied bit, not a full digit), it doesn't get rounded. When you enter 0.2, it's hex representation is 1.999999999999A * 2^(-3) Cheers, Werner 41CV†,42S,48GX,49G,DM42,DM41X,17BII,15CE,DM15L,12C,16CE |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 6 Guest(s)