FORTRAN floating point accuracy problems

[HP67]

Hi and thank you very much. I was out for a walk and I realized the problem was staring me in the face. I was hoping I could update the thread before you saw it. Yes, it's the IDINT function and it happens in DROUND as you suspected.

FORTRAN IV's IDINT returns a 4 byte integer. There is no 8 byte INT() function in FORTRAN IV.

I see three possible approaches:

1) If you can define ROUND in terms of MOD (there is an 8 byte mod function available) or some other way that doesn't require taking the integer portion with the IDINT function then we can fix the rounding routine. The magnitude of the dd.mmss or hh.mmss data itself never gets close to the 2^31 signed integer limit.

2) If you know an algorithm we could substitute for the existing IDINT function. If we can go through some process to find the integer portion we can do it in 8 byte arithmetic.

3) If I can figure out the instruction and data formats for this I should be able to write an 8 byte truncation function and I wouldn't need IDINT.

Some further updates on specific points:

(03-31-2016 12:39 PM)Dieter Wrote:  But from what I read, mixing several data types in a calculation can cause weird results.

This is generally safe if you pay attention and specify the correct constants and don't use probably the one builtin that downgrades precision, IDINT! Certainly the value of the RHS is promoted to the type of the LHS. That doesn't always help, obviously.

(03-31-2016 12:39 PM)Dieter Wrote:  R = IDINT(10.0D0 ** P + 0.5D0)
Here R is declared as a double precision real. P is an (two-byte?) integer and IDINT is said to return a 4-byte integer (i.e. something with at most 10 digits). So what is the resulting precision for R?

The exponent is a 4 byte integer- I checked earlier that the result of an 8 byte real raised to any legal variable is still an 8 byte real. That part is fine since we're using integral powers less than 2^31 here.

What is the precision of R? R is a double precision (8 byte) floating point value that was promoted from a single precision 4 byte integer :-(