FORTRAN floating point accuracy problems

[Dieter]

(05-03-2016 11:54 AM)HP67 Wrote:  Ok guys, I am almost back. Dieter, do you have an idea how to proceed now that we have your rounding function available?

Using the rounding function you could try something like this:

Code:

Function dms2dd(dms)

Const safedigits = 15  ' or in your case: 16

dd = Int(dms)

If dms > 1 Then
   k = Int(log10(dd))    ' I assume Fortran features an exact log10 function
Else
   k = 0
End If

n = safedigits - 3 - k
mmss = Round(100 * (dms - dd), n)
mm = Int(mmss)
ss = (mmss - mm) * 100

dms2dd = dd + mm / 60 + ss / 3600

End Function

Here "safedigits" is the number of significant decimals we can consider exact. That's 15 for IEEE754 DP and 16 in your case.

The algorithm essentially determines how many digits after the decimal point can be assumed exact (15 resp. 16 minus the number of digits of the integer part) and rounds the mm.ss part accordingly.

Dieter