FORTRAN floating point accuracy problems

[HP67]

Dieter, something is wrong here.

The rounding function works to some extent when I specify a smallish value for decimal digits. If I round to 4 or 5 places it seems to work better than 13. I don't know why. In any case, the results when rounded are worse overall than without the rounding calculation. In many cases, it appears to be rounding down instead of up.

I think the problem is caused by the lack of a nearest integer function but I am not sure. The IDINT function is a double precision (8 byte) that simply returns the integer part like IP in RPL.

The MMSS value is for the following line (out of order because I added a WRITE statement to the subroutine to debug it. For example in the first test the input value is 89.11150 but MMSS is extracted and rounded to 13 digits according to your example above and comes out to 11.1499999649823)

Code:


  TEST    DMS       EXPECT       RESULT         DIFF

 MMSS   11.1499999649823     
     1    89.11150    89.18750    89.18750     0.00000
 MMSS   14.9999999441206     
     2    12.15000    12.25000    12.26111     0.01111
 MMSS   29.9999999348074     
     3    33.30000    33.50000    33.51111     0.01111
 MMSS  0.299999956041574     
     4    71.00300    71.00830    71.00833     0.00003
 MMSS   24.5299999602139     
     5    42.24530    42.41470    42.41472     0.00002
 MMSS   42.2499999403954     
     6    38.42250    38.70690    38.70694     0.00004
 MMSS   30.2999999374151     
     7    29.30300    29.50830    29.50833     0.00003
 MMSS   49.4899999350309     
     8     0.49490     0.83020     0.83028     0.00008
 MMSS   14.9999999441206     
     9    75.15000    75.25000    75.26111     0.01111
 MMSS   22.2999999765307     
    10    43.22300    43.37500    43.37500     0.00000
 MMSS   33.4499999415129     
    11     9.33450     9.56250     9.56250     0.00000
 MMSS   57.5219999533147     
    12    33.57522    33.96450    33.96450     0.00000
 MMSS   7.24419993348420     
    13    13.07244    13.12345    13.12345     0.00000
 MMSS   29.9999999348074     
    14    21.30000    21.50000    21.51111     0.01111
 MMSS   47.2111999522895     
    15    59.47211    59.78920    59.78920     0.00000
 MMSS   11.0095999669284     
    16    65.11010    65.18600    65.18360     0.00240

If I specify rounding to 4 places the results improve a lot but are still worse than when unrounded:

Code:


  TEST    DMS       EXPECT       RESULT         DIFF

 MMSS   11.1500000000000     
     1    89.11150    89.18750    89.18750     0.00000
 MMSS   15.0000000000000     
     2    12.15000    12.25000    12.25000     0.00000
 MMSS   30.0000000000000     
     3    33.30000    33.50000    33.50000     0.00000
 MMSS  0.300000000000000     
     4    71.00300    71.00830    71.00833     0.00003
 MMSS   24.5300000000000     
     5    42.24530    42.41470    42.41472     0.00002
 MMSS   42.2500000000000     
     6    38.42250    38.70690    38.70694     0.00004
 MMSS   30.3000000000000     
     7    29.30300    29.50830    29.50833     0.00003
 MMSS   49.4900000000000     
     8     0.49490     0.83020     0.83028     0.00008
 MMSS   15.0000000000000     
     9    75.15000    75.25000    75.25000     0.00000
 MMSS   22.3000000000000     
    10    43.22300    43.37500    43.37500     0.00000
 MMSS   33.4500000000000     
    11     9.33450     9.56250     9.56250     0.00000
 MMSS   57.5200000000000     
    12    33.57522    33.96450    33.96444     0.00006
 MMSS   7.24000000000000     
    13    13.07244    13.12345    13.12333     0.00012
 MMSS   30.0000000000000     
    14    21.30000    21.50000    21.50000     0.00000
 MMSS   47.2100000000000     
    15    59.47211    59.78920    59.78917     0.00003
 MMSS   11.0100000000000     
    16    65.11010    65.18600    65.18361     0.00239

Do you have any idea what's going on? Why should rounding to more places cause it to round down instead of up?

Here is the code at this point:

Code:


      DOUBLE PRECISION FUNCTION DMS2DD (DMS)
      REAL*8 DMS
C
      REAL*8 DROUND
      REAL*8 DD, MM, SS
      REAL*8 MMSS
C
      DD = IDINT(DMS)
C
      MMSS = 100.0D0 * DROUND(DMS - DD, 4)
      WRITE (*,*) 'MMSS', MMSS
      MM = IDINT(MMSS)
      SS = (MMSS - MM) * 100.0D0
      DMS2DD = DD + MM / 60.0D0 + SS / 3600.0D0
      RETURN
      END

Code:


      DOUBLE PRECISION FUNCTION DROUND (X, P)
      REAL*8 X
      INTEGER P
C
      REAL*8 R
C
      R = IDINT(10.0D0 ** P + 0.5D0)
      DROUND = IDINT(R * X + 0.5D0) / R
      RETURN
      END