Post Reply 
FORTRAN floating point accuracy problems
04-14-2016, 08:18 AM
Post: #44
RE: FORTRAN floating point accuracy problems
I don't think binary arithmetic is to blame here - rather, it's the lack of a true conversion to integer. Either truncation or rounding would be fine, so long as they are accurate to all digits. The problem here is a little extra "cosmetic" rounding which seems to be difficult or impossible to avoid.

It might be OK if you take the integer part and then check by subtraction that it's the right value - now you have a program instead of a formula.
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: FORTRAN floating point accuracy problems - EdS2 - 04-14-2016 08:18 AM



User(s) browsing this thread: 5 Guest(s)