Torture tests: what do they mean?

[jebem]

This site calls my attention to how different calculators are concerning the used algorithms to get a result for a specific problem.

For example, the tan(355/226) has an ideal result of -7497258.185...
It gives me the following results (in Radian mode) for these models:

HP-Prime: -7497089.06508
HP-50G: -7497089.06508
Casio FX-991DE PLUS: -7497258.44
WP-34S: -7497258.194

Clearly the HP models algorithms deviates somehow from the expected result, contrary to what the Walter&Pauli algorithms which are giving fantastic close results.
My new acquisition, the Casio FX-991DE PLUS, is a nice surprise with a close second in this torture test, probably because it uses 15 digits internally like many others Casio calculators.

[Paul Berger (Canada)]

On a prime in CAS mode I get -7.49725847868E6.

[Paul Dale]

To toss a spanner in the works a little. The 34S is getting the correctly rounded result for its precision.

Code:

    Formula                       Exact result                                                      16 digit rounded result
355/226                         1.570796460176991150442477876106194690265486725663716814159292...         1.570796460176991
tan(1.570796460176991)   -7497258.1937817903530300676383500305596348743875526813252...             -7497258.193781790

This final number is what the 34S is giving, the correctly rounded sixteen digit result that was asked for.


- Pauli

[DavidM]

(05-13-2014 10:58 PM)jebem Wrote:  For example, the tan(355/226) has an ideal result of -7497258.185...
It gives me the following results (in Radian mode) for these models:

HP-Prime: -7497089.06508
HP-50G: -7497089.06508
Casio FX-991DE PLUS: -7497258.44
WP-34S: -7497258.194

Clearly the HP models algorithms deviates somehow from the expected result, contrary to what the Walter&Pauli algorithms which are giving fantastic close results.

As Pauli has mentioned, the 34s is giving the correctly rounded result to 16 digits.

Likewise, I believe you will find that the Prime/50g results listed above are the correctly rounded results to 12 digits for a 12-digit input parameter.

In other words, the results are not for the exact value of 355/226, but rather 355/226 rounded to x digits.

For what it's worth, a SysRPL program using 15-digit extended reals on a 50g gives a result of -7.49725824999067E6 (-7497258.24999067). A quick check with Wolfram Alpha shows this to be a correctly rounded result to 15 digits for the 15-digit input of "tan(1.57079646017699)".

[Paul Dale]

Using my trusty 34S and the RSD function, I've produced a table of the corectly rounded results for differing numbers of significant digits.

Code:

digits     355/226               tan(column 2)
8       1.5707965               -5773501.9
9       1.57079646              -7507219.88
10      1.570796460             -7507219.878
11      1.5707964602            -7495965.1067
12      1.57079646018           -7497089.06508
13      1.570796460177          -7497257.687902
14      1.5707964601770         -7497257.6879019
15      1.57079646017699        -7497258.24999067
16      1.570796460176991       -7497258.193781790
17      1.5707964601769912      -7497258.1825400143
18      1.57079646017699115     -7497258.18535045830
19      1.570796460176991150    -7497258.185350458299
20      1.5707964601769911504   -7497258.1853279747468

The Prime and 50G are both producing the correctly rounded twelve digit results. The fifteen digit extended precision on the 50G is also getting the answer correct. I've no idea what the CASIO is doing


- Pauli

[DavidM]

It appears that the original author of the "torture test" chose 355/226 due to its rough approximation of \(\pi\)/2. I wonder how much more interesting Pauli's table would be with \(\pi\)/2 as the target of the TAN function?

50g:

Code:


12-digit tan(1.57079632680) = -195948537906
15-digit tan(1.57079632679489) = 1.51074943811517E14

I'd be especially curious as to what the Casio results are for that.

This all seems very reminiscent of a recent post in the HP Prime section. While I'm not trying to re-hash the issues brought up in that discussion, I think it does point out the importance of understanding the implications of guard digits, precision, and how a given calculator displays full values vs. rounded representations of internal numbers.

[Dieter]

(05-13-2014 10:58 PM)jebem Wrote:  For example, the tan(355/226) has an ideal result of -7497258.185...
It gives me the following results (in Radian mode) for these models:

HP-Prime: -7497089.06508
HP-50G: -7497089.06508
Casio FX-991DE PLUS: -7497258.44
WP-34S: -7497258.194

Sure. They all do not calculate tan(355/226). The argument cannot be respresented exactly in 10, 12 or even 16 or 30 digits. All you can expect on a 12-digit machine is a 12-digit result for a 12-digit-rounded input, i.e. tan(1,57079646018).

Take a look at the derivative. At 355/226 it is 1/cos²(355/226) = 5,62 E+13. On an n-digit calculator, the input may be off by 5 E–n (half a ULP), which means that the tangent can by off by 2,81 E+14 · 10^–n. For a 12-digit machine this means that the result has a tolerance of ±281. In other words: you cannot even trust the three last digits left (!) of the decimal point! Your results prove this. The possible relative error is 3,7· 10^7–n, which means that 8 digits are lost and only n – 8 digits of the result can be trusted.

Quote:Clearly the HP models algorithms deviates somehow from the expected result, contrary to what the Walter&Pauli algorithms which are giving fantastic close results.

Even with a "perfect" algorithm a 10-digit calculator will only get two (!) digits right, and on an 8-digit machine the result is completely meaningless. If you want 12 valid digits in the result, you need 20 digit precision for 355/226.

We should always keep in mind that our calculators usually do not work with exact numbers. So the output cannot be exact either.

Dieter

[HP67]

(05-14-2014 05:35 AM)DavidM Wrote:  point out the importance of understanding the implications of guard digits, precision, and how a given calculator displays full values vs. rounded representations of internal numbers.

Where can one learn about this? Some of you guys seem to really understand this well and it is an interesting topic I would like to know more about.

[Paul Dale]

(05-14-2014 05:35 AM)DavidM Wrote:  I wonder how much more interesting Pauli's table would be with \(\pi\)/2 as the target of the TAN function?

Assuming the 34S calculates accurately this close to the pole:

Code:

digits  pi/2                             tan(col #2)    rounded(pi)/2          tan (col #4)
   8    1.5707963                           37320540    1.5707964                 -13660250
   9    1.57079633                        -312002417    1.57079632                147169274
  10    1.570796327                      -4875590038    1.570796327             -4875590038
  11    1.5707963268                   -195948537910
  12    1.57079632679                   204222536562    1.57079632680         -195948537906
  13    1.570796326795                -9672978957157
  14    1.5707963267949             -295790719553270
  15    1.57079632679490            -295790719553274
  16    1.570796326794897          -2626266436731868    1.570796326794896  1614905391523416
  17    1.5707963267948966         51998506188720271
  18    1.57079632679489662     -1300934329906107200
  19    1.570796326794896619     4322984121858095330
  20    1.5707963267948966192   31926755792808630288

The second column is round(pi/2). The fourth is round(round(pi)/2) but only where it is different.

The difference for fifteen digits from the above is that 1.57079632679489 isn't correctly rounded


As Deiter wrote, we're really in a steep part of the tan function here.


- Pauli

[Manolo Sobrino]

A machine that can give you x accurate digits for all the domain is not working internally with just x digits. As was discussed in another thread, the HPs are likely using 2x significant digits in their trigonometrics. The bizarre thing here is that rounding to x for the input data and the arithmetics and then performing the cordic calculations with 2x tables is considered consistent.

In this case, the Mathematica arbitrary precision results are:

12 significant digits:

N[355/226, 12] = 1.57079646018

Tan[1.57079646018`12] = -7.4971*10^6

Now, 355/226 rounded to 12 and then calculating the tangent with 24:

Tan[1.57079646018`24] = -7.4970890650760106*10^6

Rounded to 12 for output:

N[%, 12] = -7.49708906508*10^6

There you go. You need at least 19 significant digits to get that.

I don't know what Casios do here. They should get -7.4972582*10^6, yet they are calculating Tan[1.5707961934128066`16]. There must be a small flaw in their implementation of the tangent.

[Paul Dale]

HP's have three guard digits typically. They do have very carefully designed algorithms.


Pauli

[DavidM]

(05-14-2014 06:19 AM)HP67 Wrote:  Where can one learn about this? Some of you guys seem to really understand this well and it is an interesting topic I would like to know more about.

By no means would I consider myself in the category of those who understand this well, but I'm at least aware enough of it to be suspicious whenever discussions about disparate results from different platforms come up. It's no small coincidence that the "targeted values" of these functions are almost always chosen to be near points of inflection, infinity, discontinuities, extremes, etc., because differences in implementation will more often be magnified in those areas.

To me, the more subtle and somewhat philosophical issue is that of how "internal" numbers are displayed when not in an exact/CAS mode. If you divide 1 by 3 on your favorite calculator, then multiply that result by 3 again, what should the result be? There are valid arguments for and against implementations with different answers to that question. Knowing how your chosen platform responds to that type of sequence can make a difference in how you approach a solution in some cases.

(05-14-2014 06:30 AM)Paul Dale Wrote:  The difference for fifteen digits from the above is that 1.57079632679489 isn't correctly rounded

Indeed! Interesting (though contrived) example of how small differences in implementation can sometimes result in large numerical differences.

(05-14-2014 06:30 AM)Paul Dale Wrote:  As Deiter wrote, we're really in a steep part of the tan function here.

Which is no accident. We were intentionally taken there by the designer of the test.

[jebem]

(05-14-2014 05:35 AM)DavidM Wrote:  It appears that the original author of the "torture test" chose 355/226 due to its rough approximation of \(\pi\)/2. I wonder how much more interesting Pauli's table would be with \(\pi\)/2 as the target of the TAN function?

50g:

Code:


12-digit tan(1.57079632680) = -195948537906
15-digit tan(1.57079632679489) = 1.51074943811517E14

I'd be especially curious as to what the Casio results are for that.

Well, my Casio FX-991DE Plus refuses to give me an answer for that question other than this:
Tan(pi/2) = "Math.-Fehler"
This is a German edition machine, you know. So that sentence means that I did something that can not exist in this calculator small brain - what is the meaning of infinite by the way?

[Dieter]

(05-14-2014 06:20 PM)jebem Wrote:  Well, my Casio FX-991DE Plus refuses to give me an answer for that question other than this:
Tan(pi/2) = "Math.-Fehler"
This is a German edition machine, you know. So that sentence means that I did something that can not exist in this calculator small brain

This simply means "math error". So it's not a specific detailled error message. Obviously the Casio handles "pi/2" as a special case and does not translate the input to tan(1,57079632679...).

Quote: - what is the meaning of infinite by the way?

I am sure you know the meaning of "infinite". ;-) If it's the German term you're asking for: that's "unendlich".

Dieter

[Dieter]

(05-14-2014 07:47 AM)Manolo Sobrino Wrote:  A machine that can give you x accurate digits for all the domain is not working internally with just x digits. As was discussed in another thread, the HPs are likely using 2x significant digits in their trigonometrics.

Most HP calculators since the HP-91 work with three additional digits. For internal calculations, that is. These digits are not presented to the user. The input has always just the number of digits you can enter. That's the crucial point here: you would need a 20-digit input for 355/226 in order to get the output (the tangent) correct to 12 digits. As already stated, you cannot trust the last eight digits.

Quote:There you go. You need at least 19 significant digits to get that.

Let's make this 20. See my previous post and, even better, Pauli's table with all results from 8 to 20 digits. The tangent of the 19-digit respresentation of 355/226 is off by 3 units in the 12th digit.

Dieter

[Manolo Sobrino]

(05-14-2014 07:23 PM)Dieter Wrote:  Most HP calculators since the HP-91 work with three additional digits. For internal calculations, that is. These digits are not presented to the user. The input has always just the number of digits you can enter. That's the crucial point here: you would need a 20-digit input for 355/226 in order to get the output (the tangent) correct to 12 digits. As already stated, you cannot trust the last eight digits.

I don't know. Testing the 50G while trying to understand the behaviour of the Prime I found that arguments are passed to functions rounded to 12. That's what they do here, they round 355/226 to 12. If they had considered 15 digits they wouldn't have got those results. There's just no way to reproduce that while rounding 355/226 to 15... In fact, the last 8 digits are accurate for the argument rounded to 12 digits, so you can really trust them if you're looking for tan(1.57079646018). But weren't we interested in tan(355/226)? In such a case the last 8 digits are not accurate. The question is why do you round when you can work with more digits? And if you have to round, what's the point of producing an accurate output for the rounded input which is of no use?

The whole precision issue for the trigonometrics was hard to understand until I found a document from TI explaining what they did for the TI-68. Then it was clear, they were doing the same. Storing Pi to 24 digits.

You can't get more precision from just an algorithm in this case. You need additional information to calculate such a number of significant digits. It has to come from high precision data in the CORDIC tables, 15 digits just don't cut it.

I got the HP results for 19 significant digits as well, probably an effect of rounding the result to 12. But the real number is not 19 or 20, for Sin it's 12*2=24, so it's likely that for all the trigs.

I haven't tested it thoroughly in order to find where they use the guard digits. I like calculators but I have the growing feeling that there's no point in wasting much more productive time with black boxes and marketing sales pitch. I've just found too many bizarre practices that are even more bizarrely rationalised. I would never work with double precision for some things, single precision for others, and then mix them. For me, it's just a lousy way to deal with numbers that has to do with the limitations of the platform and an ill-conceived idea of accuracy.

[jebem]

(05-14-2014 06:19 AM)HP67 Wrote:  

(05-14-2014 05:35 AM)DavidM Wrote:  point out the importance of understanding the implications of guard digits, precision, and how a given calculator displays full values vs. rounded representations of internal numbers.

Where can one learn about this? Some of you guys seem to really understand this well and it is an interesting topic I would like to know more about.

+1
I'm digging the WWW now for calculator algorithms. Starting from the basics: Alan Sultan explain the CORDIC. Nice reading.
http://www.qc.cuny.edu/Academics/Degrees...ultan1.pdf

[Jim Horn]

Alan Sultan's CORDIC article is a fine introduction. It misses one important simplification, though - instead of multiplying the result of the rotations by K, you can start with (K,0) as the initial vector. Then *no* multiplication is needed other than the binary shifts and adds. This was covered in the HP Journal article on the HP35 algorithms. Pretty nifty!

[jebem]

(05-14-2014 09:39 PM)Jim Horn Wrote:  Alan Sultan's CORDIC article is a fine introduction. It misses one important simplification, though - instead of multiplying the result of the rotations by K, you can start with (K,0) as the initial vector. Then *no* multiplication is needed other than the binary shifts and adds. This was covered in the HP Journal article on the HP35 algorithms. Pretty nifty!

Very cleaver, indeed!
My digital electronics background is coming back to my mind now...
Thanks for all people sharing their knowledge and thoughts here (so far: Paul, Pauli, David, Pauli, Dieter, HP67, Manolo, Jim). I'm learning a lot from you, as I'm not a mathematician by education, but I have the basics to follow what is being said.

So, apparently the big three (HP, CASIO and TI) were using CORDIC variants in the beginning, keeping absolute secrecy (and patents?) about the inner implementation details.

But now in 2014, when we have so much more processor power (ALU and FPU) in convenient small low power packages... What algorithms are in use at hardware and software level, that combined, can support our trigonometric functions?
Do they still maintain secrecy after all this years?
Or in more than 50 years nothing was really created to beat or at least give an alternative to CORDIC algorithms?
I know that 50 years is just a time tick in science. Technology is a different matter, however.

[Paul Dale]

(05-14-2014 06:19 AM)HP67 Wrote:  Where can one learn about this? Some of you guys seem to really understand this well and it is an interesting topic I would like to know more about.

I've posted this list before but here are some good texts on numeric mathematics:

• The Algebraic Eigenvalue Problem by J. H. Wilkinson -- very in-depth and a great introduction to error analysis from when it was being invented.
• Elementary Functions by Jean-Michel Muller -- a good reference on how to implement functions in many different ways. This one covers many/all of the standard algorithms -- CORDIC, series expansions, partial fractions, polynomial approximations.
• Accuracy and Stability of Numerical Algorithms by Nicholas J. Higgam -- another great book on error anlysis.
• Matrix Computations by Gene H. Golub and Charles F. Van Loan -- pretty much what it says but one of the canonical texts in numerical analysis.
• Handbook of Mathematical Functions by Abramowitz & Stegun -- A&S is the bible for mathematical and physical functions.
• NIST Handbook of Mathematical Functions -- an updated version of A&S although the overlap is deliberately kept low.
• An Atlas of Functions by Oldham, Myland & Spanier -- lots of functions and their properties. Not really essential if you have A&S but easier reading.
• Numerical Recipes by Press, Teukolsky, Vetterling and Flannery -- doesn't get good comments from numerical analysis circles but it is easy to read and contains code that can be used with sufficient care and attention to detail.
  

Be prepared to pay anything from $500 - $1k for these in hard copy.

And before you start these read and digest all of William Kahan's writings.


- Pauli