Torture tests: what do they mean?

[HP67]

(05-14-2014 11:55 PM)Paul Dale Wrote:  I've posted this list before but here are some good texts on numeric mathematics:

Thanks, that looks like a great list. Is there not one go-to book that covers the issues of accuracy and precision in implementations? Otherwise it looks like it could take a decade to absorb all this info.

(05-14-2014 08:58 PM)jebem Wrote:  I'm digging the WWW now for calculator algorithms. Starting from the basics: Alan Sultan explain the CORDIC. Nice reading.
http://www.qc.cuny.edu/Academics/Degrees...ultan1.pdf

Thanks for the link. I have heard of this before. I can't remember where.

(05-14-2014 09:39 PM)Jim Horn Wrote:  Alan Sultan's CORDIC article is a fine introduction. It misses one important simplification, though - instead of multiplying the result of the rotations by K, you can start with (K,0) as the initial vector. Then *no* multiplication is needed other than the binary shifts and adds. This was covered in the HP Journal article on the HP35 algorithms. Pretty nifty!

Thank you. I have the HP Journals. It's been a while but I seem to remember I was disappointed but not surprised that they seemed to give a good overview but left out some important details about how things were actually implemented. I'll put this on the list to go over again at some point.

(05-14-2014 10:31 PM)jebem Wrote:  Thanks for all people sharing their knowledge and thoughts here (so far: Paul, Pauli, David, Pauli, Dieter, HP67, Manolo, Jim). I'm learning a lot from you, as I'm not a mathematician by education, but I have the basics to follow what is being said.

I have nothing to share on this topic but I echo your thanks to all the guys on the forum including you, for sharing your knowledge.

(05-14-2014 10:31 PM)jebem Wrote:  What algorithms are in use at hardware and software level, that combined, can support our trigonometric functions?
Do they still maintain secrecy after all this years?

I think hardware implementations of decimal floating point have been a big bang that has been mostly overlooked. This seems to be something targeted at a fairly narrow range of problems but dealing with the important topic of money. There is a lot of good info at Mike Colishaw's Floating Point Decimal Pages. IBM has published a lot of information on this and the implementations are described in an architectural document. But this whole thing does not apply much to general scientific computation as far as I know since there are already binary and hex floating point hardware implementations for those topics and obviously IEEE binary FP is the main target for the majority of software implementations. Still, it's interesting to note all the developments and that things are moving right along in many areas. There is a lot of doc and some important software here like the arbitrary precision decimal reference implementation with full doc on the algorithms. Even IBM seems more open these days.

(05-14-2014 10:31 PM)jebem Wrote:  I know that 50 years is just a time tick in science. Technology is a different matter, however.

That seems to be the issue and we talked about it in other threads. The software and the advances in science behind the software have not kept pace with the advances in technology (hardware). As you alluded to, this is just the way things seem to work and nobody seems to be able to get around it.

[Paul Dale]

If I had to suggest one resouce, I'd go for William Kahan's writings as a good general introduction. I can't suggest just one of the books I referenced.

There are many PhDs still waiting in this area, so yes a decade might be a decent start

Mike Colishaw's Floating Point Decimal Pages pages link to his decimal floating point library which is the basis of the 34S's numerics. There actually has quite a bit of recent development work on hardware decimal floating point arithmetic. Several processors are moving to support it natively. IBM's power cpus are beginning too and I believe Intel is working on this too.


- Pauli

[HP67]

(05-15-2014 11:37 AM)Paul Dale Wrote:  If I had to suggest one resouce, I'd go for William Kahan's writings as a good general introduction. I can't suggest just one of the books I referenced.

Thank you, I downloaded several papers. Looks like a great resource.

(05-15-2014 11:37 AM)Paul Dale Wrote:  Mike Colishaw's Floating Point Decimal Pages pages link to his decimal floating point library which is the basis of the 34S's numerics. There actually has quite a bit of recent development work on hardware decimal floating point arithmetic. Several processors are moving to support it natively. IBM's power cpus are beginning too and I believe Intel is working on this too.

Since I work on IBM mainframes (although in non-numerical systems) I have been aware of some of Colishaw's work since the 1970s. He is the father of the REXX language in addition to a lot of other neat stuff. Decimal floating point has been available on IBM hardware for a while (I just found an IBM hardware announcement from 2007 that mentions the 64 and 128 bit hardware support on System Z for decimal floating point, so that would be right about 7 years. I don't know if that was the first time it was supported or not, though) and IEEE binary FP has been supported since the late 1990s IIRC. All this in addition to IBM's native 64 and 128 bit hex FP from the S/360 days in the late 1960s. There is no shortage of fun stuff do to if you can afford a new z/Arch box and the software etc. etc. etc. to go with it!

I am glad to hear DFP is moving to other platforms because I have seen way too many threads on other forums and mailing lists where people are using binary floating point for money calculations and have no idea why that is usually not a good idea.

[jebem]

I can't resist myself to point out one more situation here.
How powerful hardware can be limited by the software design choices.

Today I have done two very simple tests, published everywhere, on 3 of very popular calculators among us:

Test 1:
(sqrt(6)^2)-6
So, the expected precise result should be Zero, right?

- On a WP-34S super precise RPN calculator (at max. mant. Fix 11): 0,00000000000
- On a Casio FX-991DE PLUS (at max. mant. Fix 9): 0,000000000
- What about a HP-50G: -0.00000000002
- And now the HP-Prime: -0.00000000002

Test 2:
(10^15 + 7.2 − 10^15) * 100
The expected result should be 720.

- On a WP-34S super precise RPN calculator (at max. mant. Fix 11): 720
- On a Casio FX-991DE PLUS (at max. mant. Fix 9): 0,000000000
- What about a HP-50G: 0
- And now the HP-Prime: 0

Go figure.
Do we have here at least two different schools of algorithms, or is it just a matter of limited number of internal digits on HP machines?

I see a similar behavior between the Prime and the 50G in so many situations that I guess HP have just ported the 50G algorithms. That would explain the apparent lack of precision for such a modern and powerful hardware sported on the Prime.

I know all the story about precision/accuracy and number of internal digits, and that the HP-Prime answer is correct for what it can handle, but the fact that it can't give a more precise answer than the other two machines (and the Casio costs less than 30 Euro) put me off a little with my beloved Prime.

My reasoning here is:
- If a low cost Casio can pass Test 1 with flying colors;
- When a moderate cost WP-34S, using much more limited hardware resources than the Prime, can handle much larger internal digits in order to give those two precise answers, why does the Prime sports such a apparent limited precision?
It is not for hardware limitations, I hope!

[DavidM]

(05-15-2014 06:01 PM)jebem Wrote:  Test 1:
(sqrt(6)^2)-6
So, the expected precise result should be Zero, right?

I have a 50g, but not the Prime. So my comments are limited to that platform.

I've highlighted what I believe to be a key phrase in your above test. You are expecting a precise result, but you're apparently using the calculator in the mode which (by design) gives approximate results. If you change your calculator to "Exact Mode On" (system flag 105 cleared) I believe you'll find a more acceptable result. Exact/Approximate mode can be toggled by holding down the right arrow key (without releasing), then pressing and releasing the Enter key.

(05-15-2014 06:01 PM)jebem Wrote:  Test 2:
(10^15 + 7.2 − 10^15) * 100
The expected result should be 720.

This one's similar to the above, but there's a subtlety that needs to be pointed out.

First, make sure you're in Exact/CAS mode.

Here's the tricky part: The inclusion of "7.2" in that expression acts as a signal to the expression parser that approximate values are included. This causes the entire expression to then be evaluated in Approximate mode, which in turn causes the limitation to 12 digits of precision for the whole expression. If you replace 7.2 with "72/10", I think you'll see that the results are different.

In this particular case, knowing how your various calculators operate can make a big difference in the outcome. I'm not trying to suggest that what they are doing is right/wrong/good/bad. At this point in time, it's reasonable to assume that technology has improved enough to give us more intelligent tools. But keep in mind, the technology of the 50g is now quite dated by today's standards. I believe the Prime inherited much of the same benefits/issues for floating point operations as the 50g (when not in CAS mode). Increasing the precision for non-symbolic operations would require a major architectural change for the Prime. It's certainly not impossible, but I wouldn't expect to see something like that any time soon (if ever).

So I guess I'm saying: I understand your frustration, but a little better understanding of how the tools are currently designed to work might give you a more acceptable (and predictable) experience. And there's nothing wrong with expecting more from the tools themselves. As consumers, we should always do that.

[Paul Dale]

(05-15-2014 06:01 PM)jebem Wrote:  So, the expected precise result should be Zero, right?

No, as a rule, the expected result won't be zero. See the above discussions about rounding.
Better still see Dieter's below.


- Pauli

[Dieter]

(05-15-2014 06:01 PM)jebem Wrote:  Test 1:
(sqrt(6)^2)-6
So, the expected precise result should be Zero, right?

No, not at all. Sure, the mathematically correct result is zero, but a perfectly working calculator may return this result or not. The reason simply is the fact that sqrt(6) is an irrational number so that no calculator in the world will safely return this value with a limited number of digits. All you get are the first n digits, where n=10, 12, 16 or even 34 (DP mode on a 34s).

Consider this:

sqrt(6) = 2,449 4897 4278 3178 0981 9728 4074...

The correct 12-digit value as returned by the 50G or Prime is 2,449 4897 4278, which is more than 0,3 ULP below the true result. So if this is squared, the result is not exactly 6 but 5,999 9999 9998 443... which is rounded to ...998 and thus 2 E–11 low. If the root would have been calculated as 2,449 4897 4279 (next adjacent value), its square would be 6,000 0000 0003 34... which is even 3 E–11 high. There simply is no 12-digit value for sqrt(6) which, if squared, would return exactly 6 (not even when rounded to 12 digits).

Now take another look at the sqrt(6) value. You will notice that it rounds very nicely to 16 digits, since the following 17th digit is a zero. This is the reason why the 34s even in SP mode (i.e. 16 digit working precision) happens to return sqrt(6)^2 as 6. It's not some "super precision" algorithm, it's simply the fact that the 17th digit happens to be a zero. You could do the same on any correctly working low-precision 6-digit machine: 2,44949² = 6,00000. There's no magic at work. Or try the same calculation with sqrt(5) on a 50G or Prime – you'll get a plain zero, simply because sqrt(5) happens to round very well to 12 digits.

Let's take an even closer look at this with some basic calculus. The first derivative of x² is 2x, so if the correctly rounded root can be off by 5 E–12, the square may be off by 2 · sqrt(6) · 5 E–12 = 2,45 E–11 or more than 2 ULP. Which is exactly what you see in the above results.

Quote:Test 2:
(10^15 + 7.2 − 10^15) * 100
The expected result should be 720.

This is what you will get on any calculator using 17 or more digits. So the usual 12-digit calculators won't return this result, nor does the 34s in SP mode (returns 700). In DP mode with 34 digits you'll get the 720. As you will on any other correctly working machine with at least 17 significant digits. Again: there's no magic at work. Replace the 10^15 with 10^40 and even the 34s in DP mode won't cut it. Replace it with 10^10 and any 12-digit machine will get a 720 as well. In general, if the power is 10^n, any machine with n+2 digits working precision will do the trick.

Quote:Do we have here at least two different schools of algorithms, or is it just a matter of limited number of internal digits on HP machines?

You should be able to answer this question now. ;-)

Dieter

[rprosperi]

(05-15-2014 11:01 PM)Dieter Wrote:  

(05-15-2014 06:01 PM)jebem Wrote:  Test 1:
(sqrt(6)^2)-6
So, the expected precise result should be Zero, right?

No, not at all...

Dieter

Dieter - Thanks for an excellent explanation!! I don't always fully follow all the underlying math in many of the topics you contribute to, but your reply on this one was excellent and direct and has provided much insight into this plus several other recent topics that have been discussed regarding how "accurate" these machines are. I appreciate you taking the time and effort to so eloquently explain this.

[jebem]

(05-16-2014 01:57 AM)rprosperi Wrote:  

(05-15-2014 11:01 PM)Dieter Wrote:  No, not at all...

Dieter

Dieter - Thanks for an excellent explanation!! I don't always fully follow all the underlying math in many of the topics you contribute to, but your reply on this one was excellent and direct and has provided much insight into this plus several other recent topics that have been discussed regarding how "accurate" these machines are. I appreciate you taking the time and effort to so eloquently explain this.

+1.
Thanks Pauli and Dieter for the explanations.
But will agree that it is better to have more internal digits to work with, and that was my point .

[HP67]

(05-15-2014 06:01 PM)jebem Wrote:  Test 1:
(sqrt(6)^2)-6
So, the expected precise result should be Zero, right?

As DavidM pointed out, in CAS (exact mode) you get 0.

(05-15-2014 06:01 PM)jebem Wrote:  Test 2:
(10^15 + 7.2 − 10^15) * 100
The expected result should be 720.

And you can get that on the HP 50g also, but it's not exactly obvious how to do it.

DavidM pointed out the difference in approximate and exact mode. He also noted the decimal value 7.2 causes the expression to be evaluated in approximate mode.

If you key that expression in on the 50g and use the down arrow to pull up the expression in the Equation Writer, use SIMP to simplify the expression according to the rules of algebra and you get 720

(05-15-2014 06:01 PM)jebem Wrote:  I see a similar behavior between the Prime and the 50G in so many situations that I guess HP have just ported the 50G algorithms. That would explain the apparent lack of precision for such a modern and powerful hardware sported on the Prime.

HP has a long history of intentionally using the same (good) algorithms. If not, there would be a lot of unhappy customers. Once you learn something it's good if the next generation is evolutionary, not revolutionary. As someone who works on software with a long lifetime, I agree with this approach.

As you pointed out, science is harder than technology and doesn't go at the same pace. Is it really so bad if the Prime didn't rock the world with new algorithms?

[jebem]

(05-15-2014 07:08 PM)DavidM Wrote:  Test 1:
(sqrt(6)^2)-6
I have a 50g, but not the Prime. So my comments are limited to that platform.

Thanks David.
I tried on the HP-Prime in CAS mode: answer: 0 (against -0.00000000002 in Home mode).
So, no news here, two different calculators in one package. Nice!

(05-15-2014 07:08 PM)DavidM Wrote:  Test 2:
(10^15 + 7.2 − 10^15) * 100
(...) First, make sure you're in Exact/CAS mode.

HP-Prime in CAS mode result: 400
HP-Prime Home mode: 0
Correct mathematical result: 720.
Here the WP-34S wins hands down. More digits means more probability to get a correct result. Not the other way around.

(05-15-2014 07:08 PM)DavidM Wrote:  Increasing the precision for non-symbolic operations would require a major architectural change for the Prime. It's certainly not impossible, but I wouldn't expect to see something like that any time soon (if ever).

Yes, that was my point.
And that is why I love that WP-34S machine even more! So cheap, little, tiny, almost empty hardware internals, lightweight pocket calculator, and yet so powerful firmware inside!

(05-15-2014 07:08 PM)DavidM Wrote:  So I guess I'm saying: I understand your frustration, but a little better understanding of how the tools are currently designed to work might give you a more acceptable (and predictable) experience. And there's nothing wrong with expecting more from the tools themselves. As consumers, we should always do that.

Totally agree. As I said, I was just pointing out that some calculators can give correct answers using the default setups for very common calculation examples.

Please see Dieter brilliant explanation above, where he points out that every single calculator have its limits, depending on how one push it.

The good number of digits to have would be infinite to handle any problem.
But let's be reasonable here, for the common situations I would say that double precision arithmetic is the least that every single scientific calculator should come with by default.

I love my HP-Prime, as much as any other calculator in my collection. They remind me of the old glorious gold age of modern computation.
But for real work I use computers. But it is me, as I make a living as a system engineer since the end of the 70's, so when I need to do some more complex calculation I just use one of my computers.

[HP67]

All the Saturn models use double-precision arithmetic by modern standards (64 bits).

Personally I don't believe more digits are the answer. Heuristics that detect and handle edge cases properly and better algebra for approximate calculations would be more helpful than more digits.

Do you really want a calculator with a 64 or 128 digit display? I don't think you can win this war of precision. At some point, accuracy becomes more important than precision.

[jebem]

(05-16-2014 09:03 AM)HP67 Wrote:  All the Saturn models use double-precision arithmetic by modern standards (64 bits).

Personally I don't believe more digits are the answer. Heuristics that detect and handle edge cases properly and better algebra for approximate calculations would be more helpful than more digits.

Do you really want a calculator with a 64 or 128 digit display? I don't think you can win this war of precision. At some point, accuracy becomes more important than precision.

That might be true, however it do not explain why other calculators like Casio, Texas, the WP-34S, and even some HP models like the 300S+ can handle those simple arithmetic problems by giving the correct mathematical answer. Please check Dieter explanation, it makes all the sense to me.
For me it is not about the number of digits in the display, it is about the number of internal digits used in approximate calculations.

[HP67]

The TI is not carrying 16 digits. I don't know about Casio. It would seem their algorithms make more difference than the number of digits. Dieter explained how less digits can sometimes appear to give the correct answer. Maybe somebody who knows more about the specific models you mention can explain it to us. Anyway, you said you want the correct answer, and we all agree on the value of that. I am only saying the war on precision is a war that we can't win. We have to fight the battle from the standpoint of smarter algorithms.

On the 50g we do get the correct answer to both problems, exactly as you entered them. I think the situation is not so bad

[Dieter]

(05-16-2014 09:18 AM)jebem Wrote:  Please check Dieter explanation, it makes all the sense to me.

Thank you very much. But let us not for get the most important thing here:

Quote:For me it is not about the number of digits in the display, it is about the number of internal digits used in approximate calculations.

It's all about the user. We should never forget that our calculators cannot do exact mathematics (except those with a CAS). All they do are numerical approximations. We always have to keep this in mind - and use our brains to get the most out of it.

A nice example has been discussed recently. It was about the term tan((0.5 – x)*pi) that was calculated within a WP34s function. This formula will return inaccurate results as x approaches zero, and finally the results will be completely meaningless as the tangent of the internal approximation of pi/2 is returned for any x below 5E–11, 5E–13, 5E–17 or 5E–35, depending on the number of available digits. So sooner or later this approach will fail. Now it's up to you, the user, to realize this danger of digit cancellation and use your mathematical knowlegde: rewrite the expression as cot(pi*x) and everything works fine even for the lowest x the calculator can handle. Use this for any x below 0.1 and the original formula for all others, and you'll get a correct answer.

In other words: any calculator is just as good as its user allows it. It's the same story as with any other tool. ;-)

Dieter

[DavidM]

Jose, Dieter has provided a great explanation of why calculators (any of them) have to be inexact in Test 1.

Here's a paradigm shift that may also help you better understand a related issue about calculator results: instead of thinking of your 50g as a 12-digit calculator, pretend for a moment that it's actually a 10-digit system. To help with the charade, set your display mode to FIX 9 and the operating mode to Approximate Results (ie. non-CAS).

Using this configuration, perform the following:

Code:

     6  SQRT  x^2
     (note: don't try the full Test 1, it won't help in this case :-) )

In FIX 9 mode, you should see the desired result of 6 - along with 9 0s after the fraction mark. Is the problem solved? "Of course not!" you say, "the actual number in stack level 1 is 5.99999999998, it's simply being rounded to 6 for the display mode."

I humbly submit to you that this is actually very similar to how many calculators operate. To lessen the effect of approximate results, many systems actually display an altered version of the internal number that's really present. It's a bit more complicated than the above example, but still close.

I don't have a large collection of calcs as many here do, but I do have an old TI-34 at my disposal. Here's an example of what I'm referring to on that unit:

Code:

     Calculation: 7 / 9 =
     Displayed Results: 0.777777777

If I now subtract 0.777777777 from that number, we might expect to get 0. Instead, I get a curious result:

Code:

     Calculation: - 0.777777777 =
     Displayed Results: 7.78 E-10

So, despite TI calling this a 10-digit calculator, it was obviously maintaining more than 10 digits for the result of 7/9. Also notable is that it didn't display the result as 0.777777778. It truncated the number for the display (as opposed to rounding), but obviously carried more digits anyway.

One of the reasons that I don't get too uptight about seeing these non-perfect results in approximate mode on a 50g is because I know that all calculators have them, but some of them go to great lengths to hide the "unexpected" results (and extra digits) from you. I actually prefer the way the 50g presents things: WYSIWYG. There aren't any hidden digits to the right that are being treated in a special manner. It does use extra guard digits for internal calculations, but when it presents the answer of 7/9 as 0.777777777778, that's exactly what you have. No more, no less. Is that the correct answer? Not exactly, but as close as any 12-digit calculator could provide for you.

[jebem]

Hi David,
Agree of course.
Since my days from electrical engineering school that we just needed 2 or 3 decimal places in our calculations, so it was common to use fix 2 on my hp-25. Rounding and knowing the error margin did the trick in common electronic circuits.

[Joe Horn]

(05-16-2014 08:52 AM)jebem Wrote:  

(05-15-2014 07:08 PM)DavidM Wrote:  Test 2:
(10^15 + 7.2 − 10^15) * 100
(...) First, make sure you're in Exact/CAS mode.

HP-Prime in CAS mode result: 400
HP-Prime Home mode: 0
Correct mathematical result: 720.
Here the WP-34S wins hands down. More digits means more probability to get a correct result. Not the other way around.

Please note that this wasn't really done in "Exact/CAS mode" as requested. The decimal point in the 7.2 forces Prime to evaluate the whole thing in approximate mode, not exact mode. If you really want to evaluate that expression in Exact/CAS mode, you must use 72/10 (or any other exact equivalent of 7.2, e.g. 7+1/5) instead of 7.2 with a decimal point in it. If you do that, you'll get the correct answer of 720 in Exact/CAS mode.

[Tugdual]

(05-13-2014 10:58 PM)jebem Wrote:  This site calls my attention to how different calculators are concerning the used algorithms to get a result for a specific problem.

For example, the tan(355/226) has an ideal result of -7497258.185...
It gives me the following results (in Radian mode) for these models:

HP-Prime: -7497089.06508
HP-50G: -7497089.06508
Casio FX-991DE PLUS: -7497258.44
WP-34S: -7497258.194

Clearly the HP models algorithms deviates somehow from the expected result, contrary to what the Walter&Pauli algorithms which are giving fantastic close results.
My new acquisition, the Casio FX-991DE PLUS, is a nice surprise with a close second in this torture test, probably because it uses 15 digits internally like many others Casio calculators.

Hi Jebem,
please note that the 34s has a "double" precision mode.
I do get -7497258.185325587112905071814481370
Wolfram Alpha says that the result shall be
-7.497258185325587112905071831891248663417267943785263161571223470151837884956238​9571281869682281839570540842373435... × 10^6

So the 34s could achieve 26 digits.
Quite impressive but I will not congratulate Walter or he will think I worship him...

[Tugdual]

(05-16-2014 09:03 AM)HP67 Wrote:  All the Saturn models use double-precision arithmetic by modern standards (64 bits).

Personally I don't believe more digits are the answer. Heuristics that detect and handle edge cases properly and better algebra for approximate calculations would be more helpful than more digits.

Do you really want a calculator with a 64 or 128 digit display? I don't think you can win this war of precision. At some point, accuracy becomes more important than precision.

I agree, but on the other hand, it is good to understand that calculators are delivering incorrect results. When I was a kid, I thought the machine was always right.