Numerical accuracy on the HP Prime (in Python, Home, and CAS)

[ftneek]

The main purpose of this thread is to try and gain insight about how a cosine value is computed on the Prime, Xcas, etc., discuss limitations of floating point arithmetic, constraints of the HP Prime, etc., and see if it would be possible to increase the accuracy of some operations (like cos, MOD, etc) over floating point arithmetic.

I recently noticed this thread, where it was reveled that the G2 returns an incorrect result of -Inf when evaluating cos(1e50) from the CAS View. This and this thread also had some interesting details.

It made me curious, what should the correct (or expected) result be? I decided to check some different sources, put the results into a spreadsheet, and tried to sort it in a way that made sense (to me). I’ve bolded the results from the HP Prime, results agreeing with other platforms until the last digit are highlighted in green, and results not agreeing with other platforms are highlighted in red. I’ve also attached a python script which simply prints the result of each calculation in case you’d like to try it on another platform and compare the results.

results:
  results_cos(1e50).xlsx (Size: 12.3 KB / Downloads: 20)

python script:
  cos(1e50).txt (Size: 671 bytes / Downloads: 8)

As you can see, the HP Prime’s Python App environment had 8 results that were significantly different from Python or other MicroPython ports (half of them a reported domain error), and 4 results that differed only in the last digit. I won’t complain about the last digit being different, but I’m curious to know a reason for the difference. It seems the Prime might have rounded up while the other Python environments did not.

The Home environment matches the result on Wolfram alpha when in Degrees and Gradians mode, but not Radians. Presumably the G2’s CAS should match the results from the 14425 VC, though it does not seem to be as accurate as python or Wolfram alpha, likely due to the nature of CAS being an exact environment.

Still, is it possible to do better than the current results in Home and CAS?

My first thought is that for an angle less than 0 or greater than 2*pi you can add or subtract multiples of 2*pi until it is within the interval [0,2*pi], or in other words use the congruent angle mod 2*pi. This seems to happen automatically in most other Python implementations.

Home:
13 MOD PI -> 0.43362938564 (ok)
13 MOD 2*PI -> 3.14159265359
The last result should have matched the one above it.
13 -2*(2*pi) -> 0.43362938564 (ok)
13 MOD (2*PI) -> 0.43362938564 (ok)
(1ᴇ50) MOD 2*PI -> 0
1ᴇ50 MOD (2*PI) -> 6.149975477

There is an issue when the dividend must be evaluated and is not enclosed in parentheses. A result of 3.14159265359 does not make sense if the dividend was 2 or PI, let alone 2*PI. It seems MOD is severely limited by floating point roundoff. The last MOD statement shown does not agree with Python (~3.92) or Wolfram alpha (~4.05). The Prime could probably benefit from an improved floating point MOD routine, even for fairly simple inputs. (parentheses bug).

CAS (Exact mode):
13 MOD PI -> -4*π+13 (nice)
13 MOD 2*PI -> π (should have matched above)
13 MOD (2*PI) -> -4*π+13

CAS (Exact or approx does not matter):
1e50 MOD (2*PI) -> 6.64613997892e35

The parentheses issue is present in CAS as well, but at least CAS can return an Exact answer. Unfortunately 1e50 was not reduced to a value less than 2*pi (not even just the significant digits). Maybe another improvement to consider for Home would be to return Exact MOD results in some cases if Intelligent Math is enabled.

So, how would you compute cos(1e50) on a G2 as accurately as possible? Is it possible to beat the Python results? Feel free to share your comments, ideas, programs, results from other calculators, other problems where the numerical accuracy could potentially be improved through one method or another, or explain why it wouldn’t be possible to compute with more precision.