Summation based benchmark for calculators

[AlexFekken]

(12-23-2017 02:27 PM)pier4r Wrote:  But for the objective of the test (not much accuracy, rather timing) I would suppose that whatever spread of the inputs would be more or less similar.

Incredible. We don't even seem to use the word "spread" with the same meaning... :-)

Wrt Simpsons rule, as someone has already pointed out in another thread: Simpson's rule is for student's only. Any serious, integration algorithm would be adaptive rather than using points with a fixed interval spacing. But perhaps it explains your obsession (in this thread) with summations.

My main personal suggestion for performance benchmarking, if you really, really feel that you have to create one, is that you do the exact opposite of what you are doing, i.e. the opposite of combining a bunch of ad hoc operations into a single benchmark and then hoping that your becnhmark becomes popular.

I would instead define and standardise a multitude of micro-benchmarks that test the simplest of operations in isolation (examples below). Then, on top of that, you can define your "Pier4r" aggregation benchmark(s), e.g. by taking some weighted average of the micro-benchmarks. That way others can define their own aggregated benchmarks, more suitable to their own use cases. Perhaps more importantly, it might help detect the relative strong and weak areas of each implementation (which can direct the implementers to the areas where they should look at improving things, or tap themselves on the back).

For example, to test the performance of loop summation I would simply write some code to calculate 1+1+1+1+1... etc, in a loop and another one to force non-integer arithmetic, e.g. sqrt(2)+sqrt(2)+sqrt(2)+sqrt(2)+ etc. I would experiment with, i.e. have different benchmarks for, getting the values out of variables versus using contstants as well. But I would certainly *avoid* calling transcendental functions (whose execution time would probably dominate the total execution time) and then suggest that I am testing the performance of a summation loop...

Wrt testing the performance of the transcendental functions, I would test them all (i.e. the ones you are interested in) in as much isolation as possible (e.g. separately time the overhead of the loops to set up the arguments; do *not* sum the values unless optimization forces you). And the basic trig functions already show that you may want multiple function-specific micro-benchmarks for every single function. For example, as some initial suggestions for the sine:

sin(x), x = -10..10 (0.001) in radians
sin(x), x = -1E6..1E6 (1) in radians
sin(x), x = -500..500 (sqrt(0.02)) in degress
sin(x), x = -1E8..1E8 (sqrt(20)) in degress
sin(x+iy), x = -10..10 (0.001), y = -10..10 (0.001) in radians
etc

EDIT: it might be a good idea to adjust the numbers above so as to have the same number of function evaluations per micro-benchmark. And to keep that number the same for the other functions.

Some of the thoughts (and correct or incorrect assumptions about implementations) behind the reason for having these different micro-benchmarks:
- Test a decent spread of unrelated (from the function's perspective) inputs
- Calculations for small and large arguments may perform very differently and not everyone may care about (very) large arguments (in this case) for their personalised aggregated benchmark
- Calculations in radians and degress may perform very differently (especially if the implementation has been careful to deal with rounding appropriately in each mode) and, again, people may only care about one of these modes
- Distinguish between real and complex calculations because the performance may be very different (perhaps predictably so for simple trig and exponential functions, but probably not for others, e.g. Gamma) and, again, not everyone will care about complex calculations.

Lots of words, encouraging you to do lots of work, by someone who isn't going to do that work. But well-intended nevertheless....