newRPL: [UPDATED April 27-2017] Firmware for testing available for download

[Bruno]

Thank you Claudio, now I can try and learn funny stuffs
This is very impressive all the work you've done !!! Congrats.

Let's talk about the code from now, and here is my first question (Don't worry, I'll try to not flood you !) :

Why did you use a slow O(n) switch/case structure in the library handlers instead of a fast O(1) structure like an array of function pointers ?
(remember the linktable from our old and so loved RPL)

An dirty example to be sure you understand me :

Code:


#define LIBRARY_NUMBER  1234

#define COMMAND_LIST \
    CMD(CMD1,MKTOKENINFO(4,TITYPE_NOTALLOWED,1,2)), \
    ...
    CMD(CMDN,MKTOKENINFO(4,TITYPE_NOTALLOWED,1,2))

...

// One function per exported command
void l1234_cmd1 (void) {
    ...
}
...
void l1234_cmdN (void) {
    ...
}

void LIB_HANDLER() {
    ...

    // Here is the command's array of function pointers
    void (*cmds[N]) (void) = {
        l1234_cmd1,
        ...,
        l1234_cmdN
    }

    if (OPCODE(CurOpcode) < N) {
        // Direct call to the command's function
        cmds[OPCODE(CurOpcode)](void);

        return;
    }

    // Add here the switch/case to handle system opcodes

    rplError(ERR_INVALIDOPCODE);
    
    return;
}

I do not find any 'bad' reason for that :

- the enum will generate appropriate index for an array
- the system opcodes could be handled outside the array
- ??

[Claudio L.]

(10-04-2016 12:16 PM)Bruno Wrote:  Why did you use a slow O(n) switch/case structure in the library handlers instead of a fast O(1) structure like an array of function pointers ?

Good point. Lately, all compilers automatically generate a jump table when compiling switch statements with consecutive numbers, so from a speed point of view it's almost equivalent, except you don't need to explicitly code the array or the jump.
Also, using separate functions causes additional overhead in passing arguments, setting up variables, etc. that could be shared between all cases. See for example how operators are handled on lib-twelve, where reading the arguments is common code at the beginning, the switch is done after we know more about the arguments. If we were to put this into separate functions, it would require overhead passing arguments, which we don't really need, or duplicate the argument reading code in each and every function, bloating the code, or having the argument reading on a separate function, causing even more overhead.
From a coding perspective, a switch() allows to "fall through" the next case, which is useful in many parts of the code (for example, subtraction is sometimes handled as negating the second argument and falling through the addition case).

So I think our use case is a very good fit for a switch() statement, the only drawback is that the code is less readable, but I found out that using a set of brackets { } in each case statement allows the code editor to collapse the whole statement, so it's not that bad in the end (if you use QT Creator or another good IDE, that is).

[Bruno]

(10-04-2016 01:31 PM)Claudio L. Wrote:  Lately, all compilers automatically generate a jump table when compiling switch statements with consecutive numbers, so from a speed point of view it's almost equivalent, except you don't need to explicitly code the array or the jump.

Ok, thanks, I envisioned this fact, compiler's optimizations are so powerfull !

(10-04-2016 01:31 PM)Claudio L. Wrote:  Also, using separate functions causes additional overhead in passing arguments, setting up variables, etc. that could be shared between all cases.
....
So I think our use case is a very good fit for a switch() statement, the only drawback is that the code is less readable,

As a workaround we can 'inline' the functions.

I've successfully compiled my first 'custom' ROM (Ok custom only mean that I have added the MEM command),
this worked well but the command is not displayed by the autocompletion feature, is there any place to declare this function I missed ?

extract from 'lib-65-system.c' :

Code:

#define COMMAND_LIST \
    CMD(TICKS,MKTOKENINFO(5,TITYPE_NOTALLOWED,1,2)), \
    CMD(MEM,MKTOKENINFO(3,TITYPE_NOTALLOWED,1,2)), \
    CMD(MEMCHECK,MKTOKENINFO(8,TITYPE_NOTALLOWED,1,2)), \
    CMD(MEMFIX,MKTOKENINFO(6,TITYPE_NOTALLOWED,1,2)), \
    CMD(READCFI,MKTOKENINFO(7,TITYPE_NOTALLOWED,1,2)), \
    CMD(PEEK,MKTOKENINFO(4,TITYPE_NOTALLOWED,1,2)), \
    CMD(POKE,MKTOKENINFO(4,TITYPE_NOTALLOWED,1,2)), \
    CMD(VERSION,MKTOKENINFO(7,TITYPE_NOTALLOWED,1,2))

MEM push the result on the stack :

[Claudio L.]

(10-04-2016 02:42 PM)Bruno Wrote:  I've successfully compiled my first 'custom' ROM (Ok custom only mean that I have added the MEM command),
this worked well but the command is not displayed by the autocompletion feature, is there any place to declare this function I missed ?

Did you try Alpha-hold and Up/Down? Autocomplete follows whatever order the library has, usually from last to first command. I can see from your picture that MEMFIX is shown, so lib65 is responding with suggestions.

BTW, how did you take that picture? Using newrpl-ui and cutting/pasting into a skin or using the x49gp emulator?

[Bruno]

(10-04-2016 08:04 PM)Claudio L. Wrote:  Did you try Alpha-hold and Up/Down?

Yes, and I noticed that when I type M and ME, the suggestions are not always correct or complete.
I will give you more details tomorrow (I do not have the same computer for now)

(10-04-2016 08:04 PM)Claudio L. Wrote:  Autocomplete follows whatever order the library has, usually from last to first command. I can see from your picture that MEMFIX is shown, so lib65 is responding with suggestions.

Perhaps MEM do not appear, but I think it's not particular to MEM.

(10-04-2016 08:04 PM)Claudio L. Wrote:  BTW, how did you take that picture? Using newrpl-ui and cutting/pasting into a skin or using the x49gp emulator?

I use x49gp and I take a screenshot with gnome-screenshot

[Bruno]

Ok, there is no problem at all with the autocompletion, it's simply that when I press and hold the [ALPHA] key with the mouse, then I use [UP] or [DOWN] with the PC keyboard, the [ALPHA] is released even if I still hold the [ALPHA] key.

Using [TAB] + [UP]/[DOWN] successively worked.

So MEM is correctly displayed :

[Claudio L.]

(10-05-2016 06:58 AM)Bruno Wrote:  Ok, there is no problem at all with the autocompletion, it's simply that when I press and hold the [ALPHA] key with the mouse, then I use [UP] or [DOWN] with the PC keyboard, the [ALPHA] is released even if I still hold the [ALPHA] key.

Using [TAB] + [UP]/[DOWN] successively worked.

So MEM is correctly displayed :

I had to redo the keyboard emulation pretty much from scratch for x49gp to be "compatible" with newRPL's routines. It doesn't surprise me that there's still a few glitches to resolve.

[Bruno]

Hi,

I'm still working on the MEM command to improve it's accuracy (and to improve my knowledge about newRPL internals ), and I need a little more help please.

PART 1 - MEM

I'll summarize here my understand about the memory structure, and how MEM is currently computing the User's free memory.

Firstly I needed to list all the memory regions and there respective sizes, and assess who is the owner (SYSTEM vs USER) :

SYSTEM'S MEMORY - 79KB - fixed size (See boot.c for details) :
- Volatile memory - 32KB
- Persistent memory - 47KB

USER'S MEMORY - 24KB - variable size :
- DSTK
- RSTK
- LAM
- DIRS
- TEMPBLOCKS
- TEMPOB

Secondly, I needed to determine how the user's memory is allocated:
-> Each region is allocated as a chunk of n * 4KB blocks, so for a fresh calculator, 6*4=24KB are used.

Thirdly, I needed to determine how to compute the current usage of each chunk (to be more accurate):
-> Using regions pointers

Fourthly, I needed to determine how many blocks are currently unallocated:
-> Using halGetFreePages() function dedicated to that purpose

Some research for a simple result :

Code:

    case MEM:
    {
        rplGCollect();

        BINT mem = halGetFreePages() << 12;

        mem += 4096 - ((DSTop - DStk) & 0xFFF);
        mem += 4096 - ((RSTop - RStk) & 0xFFF);
        mem += 4096 - ((LAMs - LAMTop) & 0xFFF);
        mem += 4096 - ((DirsTop - Directories) & 0xFFF);
        mem += 4096 - ((TempBlocksEnd - TempBlocks) & 0xFFF);
        mem += 4096 - ((TempObEnd - TempOb) & 0xFFF);

        rplNewBINTPush((BINT64)mem, DECBINT);

        return;
    }

But I still have some interogations :

- halGetFreePages() returns 404KB of free ram, so 108KB of memory allocated for a fresh newRPL installation,
but I've identified 'only' 103KB (79 + 24) of different memory regions, it stay 5KB of unknown memory, what am I missing ?

- My guess about the regions ownership (USER vs SYSTEM) helped me to determine for which regions MEM needs to compute the actual use,
but is it my guess right ?

Of course, any comment, correction and best practice about the code are welcome.

Successives call of MEM : (After each call, from 5 to 9 bytes of memory are used, I don't know why at the moment)


PART 2 - VERSION

I would like to implement the VERSION command, and as a simple demo I try this code :

Code:

    case VERSION:
    {
        char version[] = "newRPL alpha0.6.01";
        char copyright[] = "(c) Claudio Lapilli";

        WORDPTR versobj = rplCreateString((BYTEPTR)version, (BYTEPTR)version + stringlen(version));
        if(!versobj) return;

        WORDPTR copyobj = rplCreateString((BYTEPTR)copyright, (BYTEPTR)copyright + stringlen(copyright));
        if(!copyobj) return;

        rplPushData(versobj);
        rplPushData(copyobj);
        return;
    }

but I was unable to compile the ROM because gcc seems to do an implicit call to 'memcpy' that the linker was unable to resolve :

Code:

lib-65-system.o: In function `lib65_handler':
lib-65-system.c:(.text+0x4e0): undefined reference to `memcpy'
Makefile:497: recipe for target 'newrplfw.elf' failed
collect2: error: ld returned 1 exit status

(My guess about the origin of the memcpy is the copy of the string from the ROM to the RAM, but I'm not sure)

As a workaround, I overloaded the memcpy, and the compilation succeeded :

Code:

void *memcpy(void *dest,const void *source,int nwords) {
    memcpyb(dest, source, nwords);
    return dest;
}

Any idea about that ?

[Claudio L.]

(10-06-2016 02:22 PM)Bruno Wrote:  Hi,

I'm still working on the MEM command to improve it's accuracy (and to improve my knowledge about newRPL internals ), and I need a little more help please.

PART 1 - MEM

{ ... }

Some research for a simple result :

Code:

    case MEM:
    {
        rplGCollect();

        BINT mem = halGetFreePages() << 12;

        mem += 4096 - ((DSTop - DStk) & 0xFFF);
        mem += 4096 - ((RSTop - RStk) & 0xFFF);
        mem += 4096 - ((LAMs - LAMTop) & 0xFFF);
        mem += 4096 - ((DirsTop - Directories) & 0xFFF);
        mem += 4096 - ((TempBlocksEnd - TempBlocks) & 0xFFF);
        mem += 4096 - ((TempObSize - TempOb) & 0xFFF);

        rplNewBINTPush((BINT64)mem, DECBINT);

        return;
    }

But I still have some interogations :

- halGetFreePages() returns 404KB of free ram, so 108KB of memory allocated for a fresh newRPL installation,
but I've identified 'only' 103KB (79 + 24) of different memory regions, it stay 5KB of unknown memory, what am I missing ?

- My guess about the regions ownership (USER vs SYSTEM) helped me to determine for which regions MEM needs to compute the actual use,
but is it my guess right ?

Of course, any comment, correction and best practice about the code are welcome.

You are on the right track.

halGetFreePages() gives you a good idea of how much free memory we have, then you can "add" the few bytes free at the end of each region's pages, which you did correctly.
I have two comments:
a) Use TempObEnd instead of TempObSize. In general, for all regions the "Size" allocates some extra slack and it has a granularity to reduce the number of allocations needed from the system, so it doesn't reflect the actual use (look at the growXXX() functions).
b) Don't forget these are WORDPTR, so the difference (TempObEnd-TempOb) is in 32-bit words, not in bytes. So you need to subtract: (1024- ((TempObEnd-TempOb)&0xfff))<<2. This is true for all regions and all pointers.

I'd say this is as accurate as you can get. If you try to get too deep into which regions are what, your code may not be portable to other platforms, and MEM should be accurate no matter where it runs.

(10-06-2016 02:22 PM)Bruno Wrote:  Successives call of MEM : (After each call, from 5 to 9 bytes of memory are used, I don't know why at the moment)

Because every time you call MEM it allocates memory for the BINT you are pushing, which takes 12 bytes. Even if you clear the stack, the UNDO feature keeps the last 8 stacks (and uses a couple of extra words too to mark the snapshots), so the GC won't collect that memory. If you run it more than 8 times, the number should remain constant, as all 8 stacks would contain a single BINT.

(10-06-2016 02:22 PM)Bruno Wrote:  PART 2 - VERSION

I would like to implement the VERSION command, and as a simple demo I try this code :

Code:

    case VERSION:
    {
        char version[] = "newRPL alpha0.6.01";
        char copyright[] = "(c) Claudio Lapilli";

        WORDPTR versobj = rplCreateString((BYTEPTR)version, (BYTEPTR)version + stringlen(version));
        if(!versobj) return;

        WORDPTR copyobj = rplCreateString((BYTEPTR)copyright, (BYTEPTR)copyright + stringlen(copyright));
        if(!copyobj) return;

        rplPushData(versobj);
        rplPushData(copyobj);
        return;
    }

but I was unable to compile the ROM because gcc seems to do an implicit call to 'memcpy' that the linker was unable to resolve :

Code:

lib-65-system.o: In function `lib65_handler':
lib-65-system.c:(.text+0x4e0): undefined reference to `memcpy'
Makefile:497: recipe for target 'newrplfw.elf' failed
collect2: error: ld returned 1 exit status

(My guess about the origin of the memcpy is the copy of the string from the ROM to the RAM, but I'm not sure)

Avoid that like the plague. We don't want any compiler built-ins bloating the ROM.
The offending code is the allocation of the constant strings.

char *string = "Hello";

This allocates a write-able pointer called string, pointing to a write-able string pre-initialized as "Hello". The key here is that because we are running from ROM, the compiler knows that it needs to allocate space in RAM and copy the string there at run time.
To avoid it you need to use 'const', and actually a weird combination of double const for embedded systems:

const char const *string = "Hello";

This allocates a read-only pointer called string (const *), which points to a read-only string (const char). But the problem is in other platforms (PC and friends) the compilers don't like the double const statements and give warnings.

Anyway, the proper way to do it would be to put the strings in a ROMOBJECT, and add a .nrpl file to the project, look at how libraries incorporate RPL code in them by looking at how they link to those .nrpl files.
Static strings are a big NO in newRPL because most messages should be able to be translated without having to scan the code for strings.
It's quite simple to declare ROMOBJECTs that reside in an external RPL file. This RPL file should contain those 2 strings, which your handler simply needs to push to the stack. This way we can update that RPL file with the new version string without having to change the actual code, and if it's a separate RPL file containing exclusively those 2 strings, a build script could easily spit out build numbers automatically.

[Bruno]

Ok, thank you Claudio for all of these very useful informations.

[Claudio L.]

(10-06-2016 07:27 PM)Claudio L. Wrote:  So you need to subtract: (1024- ((TempObEnd-TempOb)&0xfff))<<2.

Sorry, a small correction here, it should be:
(1024- ((TempObEnd-TempOb)&0x3ff))<<2

[Sylvain Cote]

(10-06-2016 07:27 PM)Claudio L. Wrote:  const char const *string = "Hello";

This allocates a read-only pointer called string (const *), which points to a read-only string (const char).
But the problem is in other platforms (PC and friends) the compilers don't like the double const statements and give warnings.

Claudio,

The reason the compiler give you a warning is because "const char", "char const" and "const char const" are the same thing.
Also, in your hello example, the resulting pointer is not constant.

To really have a constant pointer to a constant data you need to use one of the following ...
const char * const string = "Hello"; // const char means constant data and * const means constant pointer
or
char const * const string = "Hello"; // char const means constant data and * const means constant pointer

Sylvain

[Claudio L.]

(10-09-2016 02:53 AM)Sylvain Cote Wrote:  Claudio,

The reason the compiler give you a warning is because "const char", "char const" and "const char const" are the same thing.
Also, in your hello example, the resulting pointer is not constant.

To really have a constant pointer to a constant data you need to use one of the following ...
const char * const string = "Hello"; // const char means constant data and * const means constant pointer
or
char const * const string = "Hello"; // char const means constant data and * const means constant pointer

Sylvain

My bad, I really meant "const char * const", though I double checked with the latest gcc and it didn't give me a warning for either "const char * const" or "const char const *", so I was wrong about that too, I should say "old compilers used to give a warning".
In any case, the correct syntax is the one you pointed out.

[Claudio L.]

This is unrelated to any previous discussions, but I wanted to take the time to publicly thank Günter for his HHC presentation on newRPL.
Downloads of newRPL demo spiked since his presentation, and here are the hard numbers to prove it (from now on called the "Günter Effect"):
https://sourceforge.net/projects/newrpl/...2016-10-11

Thanks again, Günter!

EDIT: For those who couldn't attend, here is the video of the presentation:

https://www.youtube.com/watch?v=YxN49QDHQMM

[Helix]

I have not watched the video, only the presentation file:
http://hhuc.us/2016/files/Speakers/Gunte...n_RPL.pptx
There is an example of a program, displayed on several lines:



I can reproduce that when writing the program, but not when editing it: it is always displayed on a single line. Is this feature not yet implemented, or is there a flag to view objects on several lines?

[Claudio L.]

(10-11-2016 05:46 PM)Helix Wrote:  I have not watched the video, only the presentation file:
http://hhuc.us/2016/files/Speakers/Gunte...n_RPL.pptx
There is an example of a program, displayed on several lines:



I can reproduce that when writing the program, but not when editing it: it is always displayed on a single line. Is this feature not yet implemented, or is there a flag to view objects on several lines?

Not yet implemented, because I want a full code beautifier rather than just splitting in multiple lines when it exceeds a certain width.

[Vtile]

Hey Claudio you at somepoint somewhere mentioned that are going to change how the +-key will work in adding a object to list. Are you going to just change the command attached to the physical key or swapping the ADD? and + commands. I hope the case is that you change the command attached to key as same way of behaviour of +-command is also found in handy string manipulation.

So if you swap the meaning of ADD and + -commands you end up braking the logic in system what comes to string (maybe other objects also).

[Claudio L.]

(10-15-2016 02:40 PM)Vtile Wrote:  Hey Claudio you at somepoint somewhere mentioned that are going to change how the +-key will work in adding a object to list. Are you going to just change the command attached to the physical key or swapping the ADD? and + commands. I hope the case is that you change the command attached to key as same way of behaviour of +-command is also found in handy string manipulation.

So if you swap the meaning of ADD and + -commands you end up braking the logic in system what comes to string (maybe other objects also).

The behavior of + for lists is now element-by-element addition. ADD appends an object to a list.
The + key is assigned to the + command, so it does element-by-element addition on lists, normal behavior on all other objects.

For strings, the behavior of + hasn't changed, it performs concatenation as there's no "addition" operation defined over strings.

[LinusSch]

Having loved my 50g for almost ten years now, I feel pretty confident newRPL is set to become the closest thing there will ever be to my "perfect" calculator.

An idea for a neat feature for hexadecimal input: if the menu is {"A" "B" "C" "D" "E" "F"}, swapping it into the 2nd menu could reorder this into {"D" "E" "F" "A" "B" "C"} for consistency with the layout of the numbers.

I think I barely qualify as a beginner level programmer, but one of the languages I can make use of is C++. Point me to where I can help out! (but expect extremely slow progress)

Also, hi everybody. /Linus

[Claudio L.]

(10-16-2016 09:53 PM)LinusSch Wrote:  Having loved my 50g for almost ten years now, I feel pretty confident newRPL is set to become the closest thing there will ever be to my "perfect" calculator.

An idea for a neat feature for hexadecimal input: if the menu is {"A" "B" "C" "D" "E" "F"}, swapping it into the 2nd menu could reorder this into {"D" "E" "F" "A" "B" "C"} for consistency with the layout of the numbers.

There's no "menu" planned for hex input. You can just press Alpha and the letters and numbers, which is the same number of keystrokes as selecting a menu, then the number. However, newRPL (as the 50g) is fully configurable, so the user can always have convenience custom menus like the one you proposed here.
Right now there's no way for a menu item to know if it's in the second or first menu, which would be needed for the dynamic reordering you are thinking. I may have to add a command to return the current menu item pressed or something similar to achieve that.

(10-16-2016 09:53 PM)LinusSch Wrote:  I think I barely qualify as a beginner level programmer, but one of the languages I can make use of is C++. Point me to where I can help out! (but expect extremely slow progress)

Also, hi everybody. /Linus

First thing you need to do is clone the repository and read all the documentation in hpgcc3.org. Then setup your machine to compile the sources (there's a post above where I outlined the steps to do this).
Then try to read/understand some portions of the code and play with some modifications. For example other contributors implemented the % command to start, and in other case the MEM command, something simple to get familiar with the inner workings.
If you can make something simple work, then congratulations, you graduated as a newRPL dev and your contributions will be merged into the code as you send them.
Now if you prefer not to get into C, we have a lot of work to do in organizing the menus and help for each and every command. This is plain RPL code and text strings, very minimal C is required to do this.
Slow progress is OK, each can work at his own speed as long as we don't step on each other's toes. Coding speed is not a concern, we only care about (in this order):
1) Code correctness: Bug free and compiler warning/error free.
2) Code readability
3) Speed of execution
Items 2) and 3) may occasionally swap places on speed critical sections.