Post Reply 
HP41: accuracy of 13-digit routines
09-02-2015, 06:11 AM
Post: #3
RE: HP41: accuracy of 13-digit routines
Thomas is correct. An example that's easier to key in is: \(\sqrt {5} \), which is equal to 2.236067977 499 7896964..., which the HP-41 says is 2.236067977, which shows that it did NOT round to the 13th digit, but truncated at the 13th digit, then rounded THAT to 10 digits.

This "truncate, then round" procedure was not only used by the old 10-digit models, but is still used by the newer 12-digit HP calculators, which calculate internally to 15 *truncated* digits, then round that result to 12 digits.

<0|ɸ|0>
-Joe-
Visit this user's website Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: HP41: accuracy of 13-digit routines - Joe Horn - 09-02-2015 06:11 AM



User(s) browsing this thread: 1 Guest(s)