FORTRAN floating point accuracy problems

[HP67]

(03-30-2016 03:41 PM)Dieter Wrote:  Yes, I think I know what's going on here: Your Fortran seems to support a precision of about ten decimal digits. That's why rounding to 13 digits does not help. I think your DROUND function is fine, just replace the 4 digit to 8 digit rounding. Or, if you are sure that nothing beyond 1/100 second occurs, simply set it to 6 digit rounding.

According to the doc it's 14 hexadecimal digits which is supposed to be approximately equivalent to 16.8 decimal digits. I don't understand how it's encoded yet so I can't verify this. But it's IBM and the doc is never wrong. It's just not very helpful in this manual. I have another couple places I want to check. So, the 13 digits you mentioned should have been perfect.

(03-30-2016 03:41 PM)Dieter Wrote:  You may test this easily: simply write a short program that prints a periodic value like 1/3 or 1/9 with 15 digits. For instance, simply let the program print 1.0D0/3.0D0 with 15 decimal places and see what you get. If it's something like 0,33333 33334 8855... you know that 31 bits or 9-10 decimal places is all you got. ;-)

Yes, and more to the point I'll run through a loop printing the output of the rounding function for 1/3 for 1..16 digits. [face palm] why didn't I think of this..

(03-30-2016 01:10 PM)HP67 Wrote:  If I specify rounding to 4 places the results improve a lot but are still worse than when unrounded:

(03-30-2016 03:41 PM)Dieter Wrote:  Sure. Rounding to 4 places will work fine if 0.mmss does not have more than 4 digits, i.e. no fractions of a second. And indeed for these cases your results are fine, only the "EXPECT" values are wrong again: dms2dd(71,0030) = 71,0083333.... and dms2dd(0,4949) = 0,83027777...., or dms2dd(65,11010) = 65,18361111... which is exactly what your program returns. The "expected" values 71,0080, 0,83020 and 65.18600 are WRONG!

They should be correct in the latest run. I checked all the calculations on an HP 48 and made a few changes before I started working on this today. I think the problem is lack of precision in the printout and the expected values are really correct but I'll read your post again and check it against my test. I'm only displaying 5 decimal places and the output routines round. I'll extend the size of the input values in the next printout. I was only using 5 decimal places because that will be enough for my end use of this routine. For debugging we should probably show more.

(03-30-2016 03:41 PM)Dieter Wrote:  Even those with 5 places (i.e. fractions of a second) are exact if the argument is rounded to full seconds (4 digits). For instance, dms2dd(13,07244) actually evaluates dms2dd(13,0724), so mm.ss = 7,2400000 and the result for this is 13,12333. Adjust the roundoff function to 5 places (or 6, or 8) and these will return exact results as well.

I don't think this is working. Although the input, expected, and actual results are shown they are rounded by the output routines. The difference value is valid at least to 5 decimal places rounded. In many cases the difference is non-zero within that precision so I think there is a still a problem.

(03-30-2016 03:41 PM)Dieter Wrote:  Now please try rounding to 6 or 8 digits and report what you got.

Yes, I'll run the test and post again tomorrow hopefully.

(03-30-2016 03:41 PM)Dieter Wrote:  And, more important, check your results with a reliable reference, e.g. an HP. ;-)

If I had a fullsize keyboard on my HP calcs I would probably throw my PC out the window!

Thank you very much for helping me with this. It's so frustrating not to get it when it comes to math.