Post Reply 
WP 31S/34S: H.MS+/H.MS- anomaly?
07-20-2015, 07:16 AM
Post: #9
RE: WP 31S/34S: H.MS+/H.MS- anomaly?
(07-20-2015 06:09 AM)Dieter Wrote:  I see that rounding is a problem with certain H.MS operations, but in this case 18 minutes are exactly 0.3 hours, and this minus 1 hour is exactly –0.7 hours = –42 minutes. So I wonder where a roundoff error may occur.

The conversion of 18 minutes came out as 0.3000....0001 in the internal reals. I didn't check where the error came from but I suspect the multiplication by 1/60 is the culprit. This was chosen because it is faster than division by 60 but it will round differently. Of course, I could be completely wrong here.

- Pauli
Find all posts by this user
Quote this message in a reply
Post Reply 

Messages In This Thread
WP 31S/34S: H.MS+/H.MS- anomaly? - 2far - 07-17-2015, 10:50 AM
RE: WP 31S/34S: H.MS+/H.MS- anomaly? - Paul Dale - 07-20-2015 07:16 AM

User(s) browsing this thread: 1 Guest(s)