A new RPL firmware for the 50g

[Claudio L.]

Hello,
I've seen this idea in several separate threads, but it always ends in "too difficult" or more recently "there's already RPL calculators in the market, so why bother".
True, but that also means we have the hardware readily available, ready to use and now (thanks to the Prime) also at a very reasonable price.
Is the community ripe for a custom "community" firmware for the 50G hardware? It's not so far fetched as many would think.
We have the technology to create modified ROM's, so how about one that's completely written from scratch?
Most of the hard coding part is done (provided we are running on the HP50g hardware):
Keyboard and screen drivers: done and in grayscale
CPU clock, exception handlers, IRQs: done and very stable
SD card driver: done but could be modified to be SDHC compatible

So let's think of the capabilities it should have (just dreaming for now):
* UserRPL source code compatibility with 50g would be a must. Existing code would need only to be recompiled in-calc to run on the new firmware. This implies basically 2 things: a) We need to write an RPL compiler/decompiler (almost trivial) and an RPL execution environment (not so trivial, especially memory management, GC, etc.), b) We need to write "clean room" implementations of the 800+ commands in the AUR.
* SystemRPL source code compatibility: Mmmmm... this is much harder, since now we need to provide "clean room" implementation for around 7000+ symbols in Carsten Dominik database.

So, just to get an opinion: if someone (as in ME and anyone out there willing to help me) were to write the RPL execution environment and a nice user interface. Will members of the community be willing to participate and help develop the 800+ functions? Many of those are really trivial, but some require extensive knowledge of the algorithms.
It wouldn't need to be in any particular language, I'd say as long as we get some pseudocode that can be understood by me (or other people helping me), it can be translated to C.
How many people would want to get involved in the main development? (need knowledge of C). If there's enough manpower, I can get the project started, but this is not something that can be done by one person alone, it needs a real community behind it.

Is there really people willing to participate?

Claudio

[Thomas Klemm]

(12-31-2013 03:04 AM)Claudio L. Wrote:  It wouldn't need to be in any particular language, I'd say as long as we get some pseudocode that can be understood by me (or other people helping me), it can be translated to C.

Compare translating this comment:

Code:


* Q := y^2/(sqrt((|x|+1)^2 + y^2) + (|x|+1))
* R := sqrt((|x|-1)^2 + y^2) +||x|-1|
* S := y^2/R if R<>0, 0 otherwise
* M := Q+R if |x|>=1, Q+S otherwise
* P := Q+S if |x|>=1, Q+R otherwise
* B := 2*x/(M+2)
* C := sqrt((P/(M+2))*(2-(P/(M+2))))
* sg(y,x) := sgn(y) if y<>0, -sgn(x) otherwise
* IM := sg(y,x)*lnp1((M/2) + sqrt((M/2)*((M/2)+2))) (sign replacement)
*
*        { arccos(B)            |B| <= (1/sqrt(2))
* RE1 := { arcsin(C)            |B| > (1/sqrt(2)) and B >= 0
*        { pi - arcsin(C)       |B| > (1/sqrt(2)) and B < 0
*
* RE2 := { arcsin(B)            |B| <= (1/sqrt(2))
*        { sgn(B)*arccos(C)     |B| > (1/sqrt(2))   (sign replacement)
*

to Python:

Code:


def sg(y,x):
    return sgn(y) if y != 0 else -sgn(x)

def inverse(x, y):
    Q = y**2/(sqrt((abs(x)+1)**2 + y**2) + (abs(x)+1))
    R = sqrt((abs(x)-1)**2 + y**2) +abs(abs(x)-1)
    S = y**2/R if R != 0 else 0
    M = Q+R if abs(x) >= 1 else Q+S
    P = Q+S if abs(x) >= 1 else Q+R
    B = 2*x/(M+2)
    C = sqrt((P/(M+2))*(2-(P/(M+2))))    
    IM = sg(y,x)*log1p((M/2) + sqrt((M/2)*((M/2)+2))) # sign replacement
    RE1 = acos(B) if abs(B) <= 1/sqrt(2) else asin(C) if B >= 0 else pi - asin(C)
    RE2 = asin(B) if abs(B) <= 1/sqrt(2) else sgn(B)*acos(C) # sign replacement
    return IM, RE1, RE2

And now imagine doing the same with SystemRPL:

Code:


:: C%>%% 2DUP DUP %%*SWAP %%ABS DUP %%1+ DUPDUP %%* 4PICK %%+
   %%SQRT %%+ 3PICKSWAP %%/ OVER %%1 %%- %%ABS DUPDUP %%* 5PICK
   %%+ %%SQRT %%+ 4ROLLOVER DUP %%0<>
   ITE
   %%/
   :: 2DROP %%0 ;
   UNROTOVER %%+ UNROT %%+ ROT %%1 %%< ?SKIPSWAP DUP %%2 %%+ ROTOVER
   %%/ %%2 OVER %%- %%* %%SQRT 5PICK DUP %%+ ROT %%/ ROT %%2 %%/ DUP
   %%2 %%+ OVER %%* %%SQRT %%+ %%LNP1 5ROLL 5ROLL DUP %%0=
   ITE :: DROP %%0< ; :: SWAPDROP %%0>= ;
   ?SKIP %%CHS
   UNROTDUP
   %%ABS %%.7 %%<= case :: SWAPDROPDUP %%ACOSRAD SWAP %%ASINRAD ;
   SWAPDUP %%ASINRAD SWAP %%ACOSRAD %%ABS
   ROT %%0>= ?SEMI
   %%CHS %%PI ROT %%- SWAP
;

Without the comments in the source code I don't think I'd be able to figure out what this piece of code does.
Others might be fluent in SystemRPL but I'm not.

Cheers
Thomas

[Paul Dale]

Isn't the purpose of implementing RPL so that many other functions can be written using RPL? That is the purpose of system RPL. I agree with Thomas, though the comments are vital. Still, they are just programming languages.

Having said all that, I think you are underestimating the difficulty of writing the 800+ functions. Writing accurate and reliable numeric code is hard. Emulating the odd behaviours in existing functions will be fun too -- I've a recollection that free42 located a number of bugs in the HP 42S which had to be emulated.

The low level hardware driving is the easy part to my mind. At least the fun doing the GC is recognised -- calculators require a compacting collector, not just the free all unused memory style collector or reference counting collectors.


- Pauli

[David Hayden]

I've created an RPL environment and implemented about 1,000 SysRPL entries. It includes garbage collection and a torture test that helps to find memory-related bugs. The code is written in C++ and SysRPL. So that might be a good place to start.

I'm going for "high compatibility" but not 100%. The simple fact that the "RPL32" uses byte-addressable memory makes 100% compatibility nearly impossible. Sizes are in bytes instead of nibbles. HEX strings must be in whole bytes, not nibbles.

Right now it's mostly "just for fun" but I have in the back of my mind the idea of implementing the HP 48 functionality on the HP 50g hardware.

As Pauli pointed out, getting the math functions right is very hard, and since RPL involves so much more than just math, I'm playing fast and loose with the it for now: real numbers are binary, not BCD and I'm using the compiler's math library. My thinking is that this code could be replaced by someone more knowledgeable than me and I'm trying to keep that in mind as I write it. For example, the formatting routine should be pretty easy to convert.

As for speed, there was a mini-challenge here recently and I wrote a solution in SysRPL. Comparing the code on the 50g emulator on my PC running at full speed, RPL32 ran about 200 times faster. These results are better than I expected so I'm not sure I trust them. At some point I'll try to compile the system in HPGCC and run it on the 50g hardware itself.

Dave

[Claudio L.]

@Thomas:
What we need to get right is the algorithms, so the initial pseudocode you showed is perfect for me. Translation to any language is not a big deal, that's why I wasn't asking for a final implementation in a particular language (much less in RPL without any comments).

@Paul:
I'm not underestimating the difficulty, I'm well aware of the complexity that's why I believe it has to be a community effort to provide good math code. No single person can do it alone (although David got pretty far all by himself...)
Now regarding emulation: I'm not talking about emulating anything.. Just creating an RPL machine, compatible at source level (to the best we can), as in the functions should have the same name (and if possible take the same arguments), but replicating undocumented or buggy behavior is not the intent here. The basic functions don't need to be written in RPL, the idea is that the user can expand the system with RPL functions, but most of the base should be optimized C or C++ code.
The way I envision it is a completely NEW system, just based on a very similar language for ease of portability and with a familiar stack interface.

@David:
I was actually counting on your reply, as I've been following your progress. I'm not sure how your implementation would fit in only 512 kbytes of RAM, but as you said, it's a good starting point. You also have a good idea of the complexity since you've already faced the challenge. So are you willing to collaborate in a project like this?

[Thomas Klemm]

(01-01-2014 02:15 AM)Claudio L. Wrote:  What we need to get right is the algorithms, so the initial pseudocode you showed is perfect for me.

We have access to the HP-48 ROM and thus to the algorithms written in SystemRPL.
I just tried to illustrate how difficult it is to understand that without the comments.
Ask HP to release the source code.

Cheers
Thomas

[Han]

I really don't see the point of re-implementing SysRPL. The whole point of SysRPL was that it served as an intermediate layer between assembly code and UserRPL in order to not have to code every individual command in assembly. However, if you're going to use C (as an example) to create commands that will replace the UserRPL ones, then there is no point in preserving the underlying SysRPL layer. Instead, you would have hooks for C code in its place. This will mean a lot of SysRPL code will no longer be compatible. However, I think the tradeoff is for the better -- SysRPL code for much more legible C code.

What is still needed is some common library code as I am sure that many of the UserRPL commands will have overlapping code. For example, one method for solving a linear system involves Kramer's Rule, which uses determinants. The best way to see what sort of core math library is necessary would be to decompile the existing ROM. If anything, this would actually be more useful in general. A community-driven disassembly of the current ROM with comments could not only preserve the algorithms but also enable you to readily implement the same algorithms in whatever language you prefer. It also opens up paths to better algorithms since more people will have their eyes on the code. Such code could serve to teach those interested in calculator development about the link between software and hardware. And lastly, long-standing bugs can finally be fixed for those who want to remain as close to the "official" firmware as possible.

Another route would be to simply import existing (free) libraries and build from scratch (much more difficult and time consuming).

As for HP releasing the source -- I have raised this question myself many times, and the answer has always been no. Not even for the HP48 series, because its source is the base for the HP50G, which is still being sold.

Just out of sheer curiosity -- are there a lot of folks who know ARM assembly programming?

[Claudio L.]

(01-01-2014 06:39 AM)Han Wrote:  I really don't see the point of re-implementing SysRPL. The whole point of SysRPL was that it served as an intermediate layer between assembly code and UserRPL in order to not have to code every individual command in assembly. However, if you're going to use C (as an example) to create commands that will replace the UserRPL ones, then there is no point in preserving the underlying SysRPL layer. Instead, you would have hooks for C code in its place. This will mean a lot of SysRPL code will no longer be compatible. However, I think the tradeoff is for the better -- SysRPL code for much more legible C code.

I agree. It would only be useful if we had an automatic "library re-compiler", that could take libs from hpcalc.org and make them run in the new firmware, so that we get a lot of existing user code working out of the box.
But that's way too ambitious at this point, and would tie us to almost an "emulation", which we don't want to do.

(01-01-2014 06:39 AM)Han Wrote:  What is still needed is some common library code as I am sure that many of the UserRPL commands will have overlapping code. For example, one method for solving a linear system involves Kramer's Rule, which uses determinants. The best way to see what sort of core math library is necessary would be to decompile the existing ROM.

Allow me to stop you right there. If we reverse-engineer the ROM, then it's not a clean room implementation and subject to copyright issues and lawsuits from HP.
The idea is to get the Advanced User guide, and provide similar functionality based only on publicly available information. Copying algorithms is unacceptable.
Besides, that will also help keep our minds fresh, avoiding to repeat past design decisions that may no longer apply to current hardware (4-bit addressing, 5-nibble numbers, etc).

(01-01-2014 06:39 AM)Han Wrote:  If anything, this would actually be more useful in general. A community-driven disassembly of the current ROM with comments could not only preserve the algorithms but also enable you to readily implement the same algorithms in whatever language you prefer. It also opens up paths to better algorithms since more people will have their eyes on the code. Such code could serve to teach those interested in calculator development about the link between software and hardware. And lastly, long-standing bugs can finally be fixed for those who want to remain as close to the "official" firmware as possible.

You are thinking more in the lines of creating an open source version of the existing ROM. I'd rather have a new system that cannot be legally challenged, and where we can make true architectural decisions. And in the future it could be ported to run on different hardware (should HP retire the 50g).

(01-01-2014 06:39 AM)Han Wrote:  Another route would be to simply import existing (free) libraries and build from scratch (much more difficult and time consuming).

As for HP releasing the source -- I have raised this question myself many times, and the answer has always been no. Not even for the HP48 series, because its source is the base for the HP50G, which is still being sold.

I wouldn't count on that.

(01-01-2014 06:39 AM)Han Wrote:  Just out of sheer curiosity -- are there a lot of folks who know ARM assembly programming?

ARM assembler, I'd think not so many (one here!), but it can be written in C/C++, so we have plenty of very capable people out there. I can handle the low-level part myself.

[David Hayden]

There are two advantages to supporting SysRPL. First, if we write some of the code in SysRPL then it will be MUCH smaller than equivalent C/C++ code. This will matter if we want to put it on hardware. Second, it means that a translator from existing HP libraries to "new RPL" is possible and that would be a big help in porting existing SysRPL to the new system.

Keep in mind also that, even if we don't implement the SysRPL entries, we'll still have to write all/most of the functionality in them.

Claudio, I'm honored that you're following my progress. I'm certainly willing to collaborate on a system, but I'm thinking that it's too early to make it available. If I get bogged down in discussions and disagreements I might lose my enthusiasm. You've been through the process with HPGCC so I'm curious to hear your thoughts.

Dave

[Han]

What about the Kinpo interface? Have you found a way to completely bypass it? Or are we still dependent its broken implementation?

[Claudio L.]

(01-01-2014 04:32 PM)David Hayden Wrote:  There are two advantages to supporting SysRPL. First, if we write some of the code in SysRPL then it will be MUCH smaller than equivalent C/C++ code. This will matter if we want to put it on hardware.

In my experience with hpgcc3, this doesn't necessarily hold true (at least the MUCH smaller part). Once you have the base code, the rest becomes mostly function calls into the ROM, much like sysRPL, and with most of the functions taking arguments directly from the data stack, a BL instruction in ARM assembler is all it's needed, so 4 bytes, same as an RPL pointer, really.
The examples I worked on, they all ended up very competitive with sysRPL code. ARM code can be surprisingly dense. I know it sounds counterintuitive, I was surprised by the results.

(01-01-2014 04:32 PM)David Hayden Wrote:  Second, it means that a translator from existing HP libraries to "new RPL" is possible and that would be a big help in porting existing SysRPL to the new system.

Keep in mind also that, even if we don't implement the SysRPL entries, we'll still have to write all/most of the functionality in them.

Yes, this would be a really good thing. The big question is if it can ever work well enough to be useful. If we can only run 5% of the code in hpcalc.org then the whole effort becomes a moot point. And to be able to run 95% of the code in hpcalc.org, we'd have to have all 7000+ entries in perfect working order (and I mean "perfect"), which would probably tie us to a design that has already been maxed out by the 50g.

(01-01-2014 04:32 PM)David Hayden Wrote:  Claudio, I'm honored that you're following my progress. I'm certainly willing to collaborate on a system, but I'm thinking that it's too early to make it available. If I get bogged down in discussions and disagreements I might lose my enthusiasm. You've been through the process with HPGCC so I'm curious to hear your thoughts.

With HPGCC, there were very few disagreements and hardly ever arguments, since technical discussions usually have reasonable solutions that everyone agrees on (after all, we are all following the scientific method, aren't we?). We only faced 2 major roadblocks: a) lack of interest from the community (one thing is people cheering on a forum, a very different thing is people actually coding and helping the project), so we were basically on our own, being the main developers and practically the only users too (count how many programs in hpcalc.org use HPGCC and you'll see what I mean). And b) time. Time makes people graduate, get married, switch jobs and move on in life. If the project takes too long to complete, then it's likely to end up forgotten.
That's why either we get a few hands together and advance the project quickly, or the project is already dead.

[Claudio L.]

(01-01-2014 07:27 PM)Han Wrote:  What about the Kinpo interface? Have you found a way to completely bypass it? Or are we still dependent its broken implementation?

HPGCC already bypassed it completely. HPGCC3 even stored itself in ROM, in a way compatible with the existing system, but there really is no need at all for the old system to be present, and I mean the whole system (as in wipe out the entire flash, replace it with our custom ROM, no Kinpo, no Saturn, no HP50g).

[Paul Dale]

Thumb code is denser than ARM and often is faster too (more code in cache despite the need for extra instructions).

- Pauli

[Claudio L.]

(01-01-2014 09:44 PM)Paul Dale Wrote:  Thumb code is denser than ARM and often is faster too (more code in cache despite the need for extra instructions).

True. HPGCC3 gets about 30% savings in size with Thumb activated (measured in a not-so-scientific way with the provided examples). I couldn't measure any speed difference, though, it's probably quite minimal (at least on the 50g hardware, that is).

[Han]

(01-01-2014 08:33 PM)Claudio L. Wrote:  

(01-01-2014 07:27 PM)Han Wrote:  What about the Kinpo interface? Have you found a way to completely bypass it? Or are we still dependent its broken implementation?

HPGCC already bypassed it completely. HPGCC3 even stored itself in ROM, in a way compatible with the existing system, but there really is no need at all for the old system to be present, and I mean the whole system (as in wipe out the entire flash, replace it with our custom ROM, no Kinpo, no Saturn, no HP50g).

I guess I am interested in contributing.

[BruceH]

(01-01-2014 08:29 PM)Claudio L. Wrote:  The big question is if it can ever work well enough to be useful.

It depends on your definition of 'useful'. If you think it means achieving binary compatibility with existing code uploaded to hpcalc.org then no, it could never be useful. If like me, you want a calculator that makes it faster to *write* programs and solve problems, rather than just *run* code faster, then a new userRPL could be very useful indeed.

I think that:

• a cleanly re-implemented userRPL in ARM rather than Saturn will be very fast, negating the main reason for using sysRPL
• with the benefit of hindsight and years of community experience, there are run-time and compile-time optimisations that can be applied to userRPL to make it faster still. (Dave and I have already exchanged ideas on this by email)
• starting afresh with a blank canvas allows more modern features to be added to userRPL to make it simpler to write programs[1] and easier to add complexity when necessary[2].
  

1. Imagine you're trying to solve a problem and are 'playing around with numbers' on the stack. Suddenly you find you've got the right answer. Instead of putting the calc into 'programming mode' and then attempting to write a program that re-creates what you just did, imagine being able to review the keystrokes you entered and the stack (and memory values) as they were at the time. You 'scroll' back up through your work and 'cut' the relevant bit into a function or a program. The calc 'knows' how many arguments you used from the stack and creates an appropriate function definition. (Or it's RPL/RPN and so doesn't matter.) You then review the newly created program and remove any redundant steps that were in your original working but not required for the final version, since removing is always easier than writing.
2. The above is great for 'informal' programs. For more considered programs, a language like new userRPL can be used. Given [1] above, the approach to defining the feature set for userRPL can be changed because it no longer has to be a one-size fits all language. I have my personal preferences as to what I would like to see in there - but rather than make this post stray too far from its original point, I will limit myself to saying that binary (or even just source) compatibility with existing sysRPL is at the very bottom of my list. ;-)
   

[Thomas Klemm]

What about RPL/2?
I managed to compile that on a raspberry pi. Never had a look at the code though. It's written in Fortran and C.

Cheers
Thomas

[David Hayden]

Han and Claudio, send me your email address in a private message and I'll send you what I have. Look it over and we can decide where to go from there.

There are definitely some neat things that we can add to RPL. One that I put in already is a SysRPL SHUFFLE command inspired by the WP34S. It shuffles up to 8 stack levels, taking the input from up to 15 levels. For example,

Code:

# 333 SHUFFLE

replaces levels 1-3 with the value that was in level 3.

Code:

# 213 SHUFFLE

is equivalent to ROT.

Code:

# abcdef SHUFFLE

replaces levels 1-6 with the contents of levels 15-10.

An even more general command is REARRANGE. This takes two arguments. The first is a number that tells how many items to add (positive argument) or remove (negative argument) from the stack. The second argument is like SHUFFLE's argument, telling it which values from the old stack to copy to the new stack.

I suspect that most of the ten zillion existing stack manipulation commands could be replaced with calls to SHUFFLE or REARRANGE and rather than spending 20 minutes figuring that SWAPUNROT 4PICKDUPDUP is the sequence that you need to get the stack arranged the way you want it, you can just tell SHUFFLE or REARRANGE what you want.

Dave

[Paul Dale]

SHUFFLE and REARRANGE are nice additions, SHFL is handy on the 34S. Many years ago, I started writing a sysRPL compiler for the 28 series that attempted to detect sequences that existed as their own entries and substituted them. Thus, you only needed to write the basic stack manipulations and the compiler figured out the shortest sequence of sysRPL calls to do them, well a shorter sequence at least.

People should also check out the OpenRPN project which had a fairly complete RPL interpreter written and quite a number of the user mode functions too. The integer mode support is pretty good, attempting to emulate the 16C functionality.

Another thought that would make for a fantastically well featured calculator would be to embed a Raspberry Pi into the device, install Wolfram's mathematica (which is free on this platform for non-commercial use), and wrapper that. You get a very full featured and well tested mathematics package for essentially free. The 50g hardware possibly isn't up to this, although I suspect it could be, but the other platforms being discussed are in the right ballpark.

I am interested in this project, but probably will have my hands full with the 43S when we start implementation.


- Pauli

[David Hayden]

Something that Bruce and I talked about was an RPL optimizer. It could replace stack manipulations like Pauli suggested, along with things like built-in constants. But the real benefit would come from knowing the stack diagrams of the entries and analyzing the types and number of values on the stack. For example, if it knows that the previous execution put two REALs on the stack and the next instruction is the UserRPL "*" command then it could replace that with %* (the SysRPL command to multiply two REAL numbers for readers who don't recognize it). I'm sure there are all sorts of other optimizations that can be made.

Pauli, just having you watching our progress would be a tremendously helpful contribution. Having done this before you could keep us pointed in the right direction. Of course, using the math code from the 34S would probably help too.

Dave