FORTRAN floating point accuracy problems
|
04-03-2016, 05:21 PM
(This post was last modified: 04-03-2016 05:22 PM by Dieter.)
Post: #27
|
|||
|
|||
RE: FORTRAN floating point accuracy problems
(04-03-2016 05:17 PM)HP67 Wrote: Again, I am only carrying 6 decimal places of expected results although I am displaying 7. This means that "differences" in the 7th place actually are not present. ;-) Maybe you can update the "expect" values to seven decimals. (04-03-2016 05:17 PM)HP67 Wrote: Thanks, I'll try your suggestions. But first I'm going to try to write an assembler subroutine for DINT. Then I can use your rounding routine which would really be great. Hmmm... I would love to know whether the suggested rounding fixes the issue. Maybe you can add these little changes and report the results... ?-) Dieter |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 1 Guest(s)