Benchmark: Savage
12-03-2017, 11:40 PM (This post was last modified: 09-29-2018 05:56 PM by StephenG1CMZ.)
Post: #1
 StephenG1CMZ Senior Member Posts: 938 Joined: May 2015
Benchmark: Savage
An implementation of the Savage benchmark, in PPL and CAS:
Update: And now in Python-syntax-in-CAS too.

Stephen Lewkowicz (G1CMZ)
12-03-2017, 11:42 PM
Post: #2
 StephenG1CMZ Senior Member Posts: 938 Joined: May 2015
RE: Benchmark: Savage
Code:
  //Benchmark: Savage   //PPL V0.1   //OMITTED: "RAD" in the original  EXPORT A_SAVAGE()  BEGIN   LOCAL AA;   AA:=1;   FOR I FROM 1 TO 2499 DO     AA:=TAN(ATAN(EXP(LN(√(AA*AA)))))+1   END;   RETURN AA;  END; #CAS  A_SavageCAS1(f):=  BEGIN   LOCAL aa;   aa:=1;   FOR I FROM 1 TO 2499 DO     aa:=TAN(ATAN(EXP(LN(√(aa*aa)))))+1   END;   RETURN (aa);  END; #END  A_SavageCAS2()  BEGIN   LOCAL aa;   aa:=1;   FOR I FROM 1 TO 2499 DO     aa:=CAS(TAN(ATAN(EXP(LN(√(aa*aa))))))+1;   END;   RETURN (aa);  END;  RRR(RR)  //REPORT RESULT,RELATIVE ERROR  BEGIN   RETURN {RR,((2500-RR)/2500)};  END;  EXPORT SAVAGE()  BEGIN   LOCAL RR;   PRINT();   PRINT("Savage Benchmark");   PRINT({" PPL1: ",TEVAL(RR:=A_SAVAGE()),RRR(RR)});   PRINT({" CAS1: ",TEVAL(RR:=A_SavageCAS1("")),RRR(RR)});     PRINT({" CAS2: ",TEVAL(RR:=A_SavageCAS2()),RRR(RR)});  //RETURN {TEVAL(RR:=SAVAGED()),RR,TEVAL(RR:=SAVAGEDCAS()),RRR(RR)};  END;

Stephen Lewkowicz (G1CMZ)
12-05-2017, 10:51 PM
Post: #3
 StephenG1CMZ Senior Member Posts: 938 Joined: May 2015
RE: Benchmark: Savage
Version 0.2

Code:
  //Benchmark: Savage   //PPL V0.2   //OMITTED: "RAD" in the original  LOCAL AA;  EXPORT A_SavagePPL()  BEGIN   LOCAL AA;   AA:=1;   FOR I FROM 1 TO 2499 DO     AA:=TAN(ATAN(EXP(LN(√(AA*AA)))))+1   END;   RETURN AA;  END;  MLFOR ()  BEGIN   //LOCAL AA;   RETURN AA:=TAN(ATAN(EXP(LN(√(AA*AA)))))+1;  END;    EXPORT A_SavageMAKE()  BEGIN   //LOCAL AA;   AA:=1;   MAKELIST(MLFOR(),AA,1,2499);   RETURN AA;  END; #CAS  A_SavageCAS1(f):=  BEGIN   LOCAL aa;   aa:=1;   FOR I FROM 1 TO 2499 DO     aa:=TAN(ATAN(EXP(LN(√(aa*aa)))))+1   END;   RETURN (aa);  END; #END  A_SavageCAS2()  BEGIN   LOCAL aa;   aa:=1;   FOR I FROM 1 TO 2499 DO     aa:=CAS(TAN(ATAN(EXP(LN(√(aa*aa))))))+1;   END;   RETURN (aa);  END;  RRR(RR)  //REPORT RESULT,RELATIVE ERROR  BEGIN   RETURN {RR,((2500-RR)/2500)};  END;  //APPROX:  //I FIND NO CLEAR WINNER BETWEEN PPL AND MAKE.  //EXACT:  //I FIND CAS2 CONSISTENTLY FASTER THAN CAS1  EXPORT SAVAGE()  BEGIN   LOCAL RR;   PRINT();   PRINT("Savage Benchmark");   PRINT({" PPL1: ",TEVAL(RR:=A_SavagePPL()),RRR(RR)});   PRINT({" MAKE: ",TEVAL(RR:=A_SavageMAKE()),RRR(RR)});     PRINT({" CAS1: ",TEVAL(RR:=A_SavageCAS1("")),RRR(RR)});   PRINT({" CAS2: ",TEVAL(RR:=A_SavageCAS2()),RRR(RR)});     END;

Stephen Lewkowicz (G1CMZ)
08-29-2018, 07:37 AM
Post: #4
 StephenG1CMZ Senior Member Posts: 938 Joined: May 2015
RE: Benchmark: Savage
Anders, thank you for reporting timings for this benchmark on the Prime C and Prime G2.

Stephen Lewkowicz (G1CMZ)
09-28-2018, 07:20 PM (This post was last modified: 11-01-2018 02:36 PM by StephenG1CMZ.)
Post: #5
 StephenG1CMZ Senior Member Posts: 938 Joined: May 2015
RE: Benchmark: Savage
Version 0.3 Beta adds an implementation of the Savage benchmark using Python-syntax-in-CAS.

That specific implementation is less accurate than the PPL version, hence the Beta release in case I have mis-coded something.
Other timings should be unaffected.

Code:
 //Benchmark: Savage   LOCAL CRID:="Savage Benchmark (Prime) V0.3 Beta";   //Beta in respect of Pythonesque results  //OMITTED: "RAD" in the original:  //User must select Radians  LOCAL AA;  EXPORT SavagePPL()  BEGIN   LOCAL AA;   AA:=1;   FOR I FROM 1 TO 2499 DO     AA:=TAN(ATAN(EXP(LN(√(AA*AA)))))+1   END;   RETURN AA;  END;  MLFOR ()  BEGIN   //LOCAL AA;   RETURN AA:=TAN(ATAN(EXP(LN(√(AA*AA)))))+1;  END;    EXPORT SavageMAKE()  BEGIN   //LOCAL AA;   AA:=1;   MAKELIST(MLFOR(),AA,1,2499);   RETURN AA;  END; #CAS  SavageCAS1(f):=  BEGIN   LOCAL aa;   aa:=1;   FOR I FROM 1 TO 2499 DO     aa:=TAN(ATAN(EXP(LN(√(aa*aa)))))+1   END;   RETURN (aa);  END; def savagepysyn() :    local ii,aa    aa=1    for ii in range(1,2499):      #aa= tan(atan(exp(math.log(sqrt(aa*aa)))))+1      aa= math.tan(math.atan(math.exp(math.log(math.sqrt(aa*aa)))))+1 #END  SavageCAS2()  BEGIN   LOCAL aa;   aa:=1;   FOR I FROM 1 TO 2499 DO     aa:=CAS(TAN(ATAN(EXP(LN(√(aa*aa))))))+1;   END;   RETURN (aa);  END;  RRR(RR)  //REPORT RESULT,RELATIVE ERROR  BEGIN   RETURN {RR,((2500-RR)/2500)};  END;  //APPROX:  //I FIND NO CLEAR WINNER BETWEEN PPL AND MAKE.  //EXACT:  //I FIND CAS2 CONSISTENTLY FASTER THAN CAS1  EXPORT SAVAGE()  BEGIN   //LOCAL aa;   LOCAL RR;     PRINT();   PRINT(CRID);   PRINT({" PPL1  : ",TEVAL(RR:=SavagePPL()),RRR(RR)});   PRINT({" MAKE  : ",TEVAL(RR:=SavageMAKE()),RRR(RR)});     PRINT({" CAS1  : ",TEVAL(RR:=SavageCAS1("")),RRR(RR)});   PRINT({" CAS2  : ",TEVAL(RR:=SavageCAS2()),RRR(RR)});   PRINT({" PySyn : ",TEVAL(savagepysyn()),RRR(aa)});   PRINT(CRID);  END;

Update: Accuracy in Python solved...I had the range wrong...Corrected v0.4 follows.
Note that the Python-in-CAS code can affect indexing.

Stephen Lewkowicz (G1CMZ)
09-29-2018, 08:16 AM (This post was last modified: 11-01-2018 08:25 AM by StephenG1CMZ.)
Post: #6
 StephenG1CMZ Senior Member Posts: 938 Joined: May 2015
RE: Benchmark: Savage
Version 0.4 adds a Python-syntax-in-CAS implementation.
This corrects the bug in my earlier beta version. Now, the accuracy SEEMS better than PPL, although since less bits are used in CAS floats I presume that is a fluke of these specific calculations (bits: see http://www.hpmuseum.org/forum/thread-11484.html)
The CAS2 procedure is now exported for improved visibility from the Run menu.

Removing the "math" references present in my V0.3Beta significantly improves timings.
Indeed, the Pythonesque timings now seem faster than the PPL version.
Code:
 //Benchmark: Savage   LOCAL CRID:="Savage Benchmark (Prime) V0.4";   //OMITTED: "RAD" in the original:  //User must select Radians  LOCAL AA;  EXPORT SavagePPL()  BEGIN   LOCAL AA;   AA:=1;   FOR I FROM 1 TO 2499 DO     AA:=TAN(ATAN(EXP(LN(√(AA*AA)))))+1   END;   RETURN AA;  END;  MLFOR ()  BEGIN   //LOCAL AA;   RETURN AA:=TAN(ATAN(EXP(LN(√(AA*AA)))))+1;  END;    EXPORT SavageMAKE()  BEGIN   //LOCAL AA;   AA:=1;   MAKELIST(MLFOR(),AA,1,2499);   RETURN AA;  END; #CAS  SavageCAS1(f):=  BEGIN   LOCAL aa;   aa:=1;   FOR I FROM 1 TO 2499 DO     aa:=TAN(ATAN(EXP(LN(√(aa*aa)))))+1   END;   RETURN (aa);  END; def savagepysyn() :    local ii,aa    aa=1    for ii in range(1,2500):      aa= tan(atan(exp(log(sqrt(aa*aa)))))+1 #END  EXPORT SavageCAS2()  BEGIN   LOCAL aa;   aa:=1;   FOR I FROM 1 TO 2499 DO     aa:=CAS(TAN(ATAN(EXP(LN(√(aa*aa))))))+1;   END;   RETURN (aa);  END;  RRR(RR)  //REPORT RESULT,RELATIVE ERROR  BEGIN   RETURN {RR,((2500-RR)/2500)};  END;  //APPROX:  //I FIND NO CLEAR WINNER BETWEEN PPL AND MAKE.  //EXACT:  //I FIND CAS2 CONSISTENTLY FASTER THAN CAS1  EXPORT SAVAGE()  BEGIN   //LOCAL aa;   LOCAL RR;     PRINT();   PRINT(CRID);   PRINT({" PPL1  : ",TEVAL(RR:=SavagePPL()),RRR(RR)});   PRINT({" MAKE  : ",TEVAL(RR:=SavageMAKE()),RRR(RR)});     PRINT({" CAS1  : ",TEVAL(RR:=SavageCAS1("")),RRR(RR)});   PRINT({" CAS2  : ",TEVAL(RR:=SavageCAS2()),RRR(RR)});   PRINT({" PySyn : ",TEVAL(savagepysyn()),RRR(aa)});   PRINT(CRID);  END;
Update: See note regarding Python-in-CAS affecting indexing.

Stephen Lewkowicz (G1CMZ)
09-29-2018, 10:47 PM
Post: #7
 Albert Chan Senior Member Posts: 1,912 Joined: Jul 2018
RE: Benchmark: Savage
Hi, StephenG1CMZ

What to you mean by Python CAS float had less bits than PPL ?

Do you have some benchmark numbers ?

How does it compared to http://www.technicalc.org/tiplist/en/fil...ip6_50.pdf
09-30-2018, 07:20 AM (This post was last modified: 09-30-2018 07:42 AM by StephenG1CMZ.)
Post: #8
 StephenG1CMZ Senior Member Posts: 938 Joined: May 2015
RE: Benchmark: Savage
In the other thread, Parisse said CAS floats had 48 bits not 53 for the mantissa.
I work on the Android version, so don't have any absolute timings, but the Pythonesque version seems about twice as fast as the PPL.

It's not the same as in your reference, in which the FOR loop is followed by a division by 2500, if I read it correctly.
I don't recall that in the original. (I do a similar calculation for relative error, but outside the timed code).
My benchmark is meant to implement the original Savage, apart from:
I don't change the mode to Radians (I don't like benchmarks that leave the device in a different mode than it started in)
And I wrap the original code in a procedure call, which wasn't in the original.

Stephen Lewkowicz (G1CMZ)
09-30-2018, 11:15 AM (This post was last modified: 09-30-2018 12:03 PM by Albert Chan.)
Post: #9
 Albert Chan Senior Member Posts: 1,912 Joined: Jul 2018
RE: Benchmark: Savage
(09-30-2018 07:20 AM)StephenG1CMZ Wrote:  Pythonesque version seems about twice as fast as the PPL.

That might explained why Python version is more accurate.

CAS float = 53 bits double, truncated to 48 bits.
Because of truncation, last bit is not as good as round-to-nearest, with slight bias.
Let's say it got effectively 47 bits precision, or about 47/3.322 ~ 14 decimals

PPL is much slower, probably doing BCD math (15 internally, rounded to 12 decimals)

BTW, Python user is more comfortable counting from zero.
For 2499 loops, range(2499) look better than range(1,2500)
09-30-2018, 02:13 PM (This post was last modified: 09-30-2018 03:28 PM by Albert Chan.)
Post: #10
 Albert Chan Senior Member Posts: 1,912 Joined: Jul 2018
RE: Benchmark: Savage
Doing Savage simulation, using Python gmpy2 / pdeclib (n=2499, radian mode):

12-decimals (halfway to even) => a = 2499.99942 402, relative error ~ 2.3E-7
48-bits CAS float (simulated) ==> a = 2499.99998 195, relative error ~ 7.2E-9

RoundToNearest:
48-bits => a = 2500.00000 041, relative error ~ 1.7E-10
47-bits => a = 2499.99999 957, relative error ~ 1.7E-10
46-bits => a = 2499.99999 984, relative error ~ 6.6E-11
45-bits => a = 2500.00000 108, relative error ~ 4.3E-10
44-bits => a = 2500.00000 179, relative error ~ 7.2E-10
43-bits => a = 2500.00000 463, relative error ~ 1.9E-9
42-bits => a = 2500.00001 457, relative error ~ 5.8E-9 <== CAS float ~ 42 bits
...
37-bits => a = 2500.00032 413, relative error ~ 1.3E-7 <== 12-decimals ~ 37 bits

CAS float is worse than expected due to its biased toward zero rounding.
12-decimals do not get back 40 bits. Decimals roundings had bigger "gap", not 1 ULP ...

Does above numbers match your Android emulation ?
09-30-2018, 04:54 PM (This post was last modified: 09-30-2018 04:56 PM by StephenG1CMZ.)
Post: #11
 StephenG1CMZ Senior Member Posts: 938 Joined: May 2015
RE: Benchmark: Savage
I get:
2499.99948647, relative error 0.000000205412, for PPL, and
2499.99998195, relative error 0.00000000722, for Python-in-CAS.
And of course 2500, 0, for the exact CAS.

Stephen Lewkowicz (G1CMZ)
10-12-2018, 09:51 AM (This post was last modified: 10-12-2018 11:05 AM by StephenG1CMZ.)
Post: #12
 StephenG1CMZ Senior Member Posts: 938 Joined: May 2015
RE: Benchmark: Savage
I have noticed an interesting feature of my Pythons-in-CAS timings on the Android.

Sometimes, I see timings of about 100 ms (i.e. faster than PPL). Sometimes, I see timings of about 2.7s (i.e. comparable to exact-CAS timings). But these are not random - once I see one value, it tends to repeat (with small variations).

In contrast, the other timings show little variation.

Stephen Lewkowicz (G1CMZ)
10-12-2018, 10:46 AM (This post was last modified: 11-01-2018 08:27 AM by StephenG1CMZ.)
Post: #13
 StephenG1CMZ Senior Member Posts: 938 Joined: May 2015
RE: Benchmark: Savage
Version 0.5 is not intended to change timings, but the timings for pythons-in-CAS do seem to be getting slower.

I seemed to have to run Savage V0.4 again before seeing the shorter times in V0.5 - which has me wondering which result is the more representative.
Is anyone else seeing a problem?

Outside the timed code, PRINT can be suppressed using boolean SP, and TEVAL timings are now returned by the main program.
GETNTIMES allows a timing to be repeated, to check repeatability.

Code:
 //Benchmark: Savage   LOCAL CRID:="Savage Benchmark (HP Prime) V0.5";   EXPORT NAMES:={   "PPL       ",   "MAKELIST  ",   "PythonsCAS",   "CAS       ",   "CASinPPL  "};  //OMITTED: "RAD" in the original:  //User must select Radians  EXPORT SP:=1;//ShowPRINT  LOCAL AA;  EXPORT SavagePPL()  BEGIN   LOCAL AA;   AA:=1;   FOR I FROM 1 TO 2499 DO     AA:=TAN(ATAN(EXP(LN(√(AA*AA)))))+1   END;   RETURN AA;  END;  MLFOR ()  BEGIN   //LOCAL AA;   RETURN AA:=TAN(ATAN(EXP(LN(√(AA*AA)))))+1;  END;    EXPORT SavageMAKE()  BEGIN   //LOCAL AA;   AA:=1;   MAKELIST(MLFOR(),AA,1,2499);   RETURN AA;  END; #CAS  SavageCAS1(f):=  BEGIN   LOCAL aa;   aa:=1;   FOR I FROM 1 TO 2499 DO     aa:=TAN(ATAN(EXP(LN(√(aa*aa)))))+1   END;   RETURN (aa);  END; def savagepythonssyn() :    local ii,aa    aa=1    for ii in range(1,2500):      aa= tan(atan(exp(log(sqrt(aa*aa)))))+1 #END  EXPORT SavageCASinPPL()  BEGIN   LOCAL aa;   aa:=1;   FOR I FROM 1 TO 2499 DO     aa:=CAS(TAN(ATAN(EXP(LN(√(aa*aa))))))+1;   END;   RETURN (aa);  END;  RRR(RR)  //REPORT RESULT,RELATIVE ERROR  BEGIN   RETURN {RR,((2500-RR)/2500)};  END;  //APPROX:  //I FIND NO CLEAR WINNER BETWEEN PPL AND MAKE.  //EXACT:  //I FIND CAS2 CONSISTENTLY FASTER THAN CAS1    SPRINT(ONE)  BEGIN   IF SP THEN    PRINT(ONE);   END;  END;  EXPORT GETNTIMES(FunN,NN)  BEGIN   LOCAL II;   LOCAL TM:={};   FOR II FROM 1 TO NN DO    CASE    IF  FunN==1 THEN TM(0):=TEVAL(SavagePPL);  END;    IF  FunN==2 THEN TM(0):=TEVAL(SavageMAKE);  END;    IF  FunN==3 THEN TM(0):=TEVAL(savagepysyn());  END;    IF  FunN==4 THEN TM(0):=TEVAL(SavageCAS1);  END;    IF  FunN==5 THEN TM(0):=TEVAL(SavageCASinPPL);  END;    DEFAULT    END;   END;   RETURN TM;  END;  EXPORT SAVAGE()  BEGIN   //LOCAL aa;   LOCAL RR;   LOCAL TM:={};   IF SP THEN    PRINT();   END;   SPRINT(CRID);   TM(0):=TEVAL(RR:=SavagePPL());   SPRINT({NAMES(1),TM(0),RRR(RR)});   TM(0):=TEVAL(RR:=SavageMAKE());   SPRINT({NAMES(2),TM(0),RRR(RR)});   TM(0):=TEVAL(savagepysyn());   //PRINT(TM(O));//DEBUG   SPRINT({NAMES(3),TM(0),RRR(aa)});   TM(0):=TEVAL(RR:=SavageCAS1(""));   SPRINT({NAMES(4),TM(0),RRR(RR)});   TM(0):=TEVAL(RR:=SavageCASinPPL());   SPRINT({NAMES(5),TM(0),RRR(RR)});     SPRINT(CRID);   //MSGBOX(TM);   RETURN TM;  END;
Note: Neither V0.4 nor V0.5 compile in Beta on Android...
Version 0.2 can be compiled in Beta if "I" is changed to "i".
Update: See note regarding Python-in-CAS affecting indexing.

Stephen Lewkowicz (G1CMZ)
11-01-2018, 08:20 AM (This post was last modified: 11-01-2018 02:38 PM by StephenG1CMZ.)
Post: #14
 StephenG1CMZ Senior Member Posts: 938 Joined: May 2015
RE: Benchmark: Savage
Note: Versions 0.3Beta, 0.4 and 0.5 utilise Python-in-CAS code.