Different trig algorithms in CAS and Home?

[Joe Horn]

(01-03-2018 08:17 PM)TheKaneB Wrote:  Hi,

I couldn't find this while searching through the forum, if it has been already discussed I'm sorry for the double post.

Anyway, if I do the following function in CAS and in Home I get 2 different answers (radians mode):

Home:
TAN(355/226) = −7497089.06508

CAS:
approx(TAN(355/226)) = −7497258.47868

The CAS result is more accurate than the Home result. Anybody knows the reason why they differ?

Believe it or not, BOTH of Prime's results are as accurate as possible... IF you keep in mind how Prime rounds its internal floating point numbers in Home and in CAS (they're different).

Home uses 12-digit BCD floating-point reals. So 355/226 is internally represented as EXACTLY 1.57079646018 followed by an infinite number of zeros. PLEASE NOTE that this is NOT mathematically equal to 355/226. It's rounded off to 12 significant digits, as necessarily must happen in Home. Now if you use a high-precision calculator (e.g. HP 50g with LongFloat set to 100 digits of accuracy) you'll see that TAN(1.57079646018) is -7497089.06507601... which, when rounded to the mandatory 12 significant digits, is −7497089.06508 which is exactly what you saw TAN(355/226) return in Home. So it's the best possible answer that a 12-digit BCD calculator can get.

CAS uses 48-bit binary floating-point reals. So 355/226 is internally represented as EXACTLY 221069948522749/2^47. As above, please note that this is not mathematically equal to 355/226; it's the largest 48-bit number which does not exceed 355/226, which is the best that CAS can do. Now if you use a high-precision calculator (e.g. HP 50g with LongFloat set to 100 digits of accuracy) you'll see that TAN(221069948522749/2^47) is -7497258.47868147... which, when rounded to the Prime's maximum display of 12 significant digits, is −7497258.47868 which is exactly what you saw TAN(355/226) return in CAS. So it's the best possible answer that a 48-bit binary calculator can get.