FORTRAN floating point accuracy problems
|
04-14-2016, 08:18 AM
Post: #44
|
|||
|
|||
RE: FORTRAN floating point accuracy problems
I don't think binary arithmetic is to blame here - rather, it's the lack of a true conversion to integer. Either truncation or rounding would be fine, so long as they are accurate to all digits. The problem here is a little extra "cosmetic" rounding which seems to be difficult or impossible to avoid.
It might be OK if you take the integer part and then check by subtraction that it's the right value - now you have a program instead of a formula. |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 5 Guest(s)