Threaded Mode | Linear Mode

matthiaspaul · 10-18-2016, 11:12 PM

Trigged by a discussion in another thread

http://www.hpmuseum.org/forum/thread-703...l#pid62643

I'd like to make two little suggestions for newRPL.

1. In that other thread we discussed a bug in the implementation of long filenames in the original HP 49g+/50g firmware, which unfortunately was never fixed and therefore exists even in the latest firmware 2.15/2.16: Under certain conditions the calculator will create SFNs containing lowercase letters, which are invalid characters in the FAT filesystem. While this does not cause any problems for the calculator itself, most other operating systems won't be able to access such files (at least not via the SFN entry). Since file exchange between calculators using the original HP firmware and newRPL is a very common use case, I think, it would be worth the effort to implement a special case into newRPL to cope with such buggy SFNs:

(10-16-2016 04:57 PM)matthiaspaul Wrote: The underlying technical reason why most operating systems won't be able to access FAT files stored with lowercase SFNs is because they will upcase the filename provided by the user and compare it with the filename stored on disk while assuming that it is stored in upper case already - consequently, SFNs stored in lower case will never match and thus the files are never found.

In many systems the upcase vector for codepoints 00h..7Fh is hardwired assuming an ASCII character set, so characters 61h..7Ah are just translated into 41h..5Ah. However, for codepoints 80h..FFh the upcase translation can differ significantly depending on the active codepage, country settings, and operating system. [...]

Knowing that the 49g+/50g firmware can or could create such invalid SFNs, perhaps newRPL should be enhanced by adding a little special case which would upcase 61h..7Ah in on-disk SFNs before filename comparisons in order to improve compatibility at least for direct file exchange. This would consume only a couple of bytes and not be conflictive with normal file access as these characters cannot normally occur in SFNs.

2. In the other thread we also discussed why it sometimes makes sense to create a LFN entry even if the filename would be short enough to fit into the 8.3 scheme and the only characters above > 127 are uppercase letters:

(10-17-2016 04:47 PM)Claudio L. Wrote:
(10-16-2016 07:26 PM)matthiaspaul Wrote: :3:"SPECIÄL" STO no reason to create a LFN entry but some system may do
There's one reason: The character >127. Creating an LFN is the only way to store it as Unicode. I modified newRPL to always create LFN's in cases like this, just to eliminate the ambiguity of using CP translation. Technically not needed if you are willing to put up with old CP mess, though.

I would like to suggest to extend this idea to the four special cases utilizing the two "case"-bits in the SFN discussed previously:

(07-28-2016 02:52 AM)Claudio L. Wrote:
(07-27-2016 08:56 PM)matthiaspaul Wrote: Yes, I know, there are also a number of other special cases:
[...]
- If a filename fits into the 8.3 scheme and either contains only lowercase letters or combines a lowercase filename and an uppercase extension or vice versa, the creation of an LFN can be suppressed as well. In this case only an SFN is created and the case information is stored in bits 4 and 3 at offset 0x0C in directory entries, so that the LFN can be recreated from the SFN later on.
[...]
All cases above are implemented exactly as described in newRPL.

IIRC (without looking at the sources again), newRPL maintains these four special cases on reads and writes. That's perfectly fine, but I would like to suggest to still create LFNs on writes if lower-case or special characters are used (or make this configurable on some "special-feature" mount flag). (IIRC this is the default under Linux.)

The reason for this is that many VFAT LFN implementations simply do not support these special bits (probably because this was not part of the original specification, was badly documented and is only supported in the NT line, not by Windows 9x). Without LFNs, this can have the effect that the case of shorter filenames is not always preserved by some platforms which might be used for file exchange. While only a cosmetical issue for many use cases under Windows and DOS, it can cause problems in conjunction with environments distinguishing between names only differing in their case. Now, the FAT filesystem is case-insensitive and newRPL works around such name conflicts using the "semicolon trick", sometimes not preserving the case could still break things, if the names are further processed.

As the code is already there, this looks like an easy change only consuming a few bytes to me. If you really want to be fancy, you could make this a two-stage effort: If (under the described conditions) creating the LFN fails (because there are no free directory entries any more), the system could fall back to only use the "case"-bit method. This way, it would consume additional directory entries only when they are still available.

Greetings,

Matthias

Claudio L. · 10-19-2016, 03:01 AM

(10-18-2016 11:12 PM)matthiaspaul Wrote: Knowing that the 49g+/50g firmware can or could create such invalid SFNs, perhaps newRPL should be enhanced by adding a little special case which would upcase 61h..7Ah in on-disk SFNs before filename comparisons in order to improve compatibility at least for direct file exchange. This would consume only a couple of bytes and not be conflictive with normal file access as these characters cannot normally occur in SFNs.

newRPL already does!
I should clarify that when you select the case-sensitive mode (default) there's no need. If there's a SFN with lowercase, the comparison will only match with lowercase letters in the search pattern, so it actually works as expected.
When case insensitive is selected, newRPL converts both:

Code:

// CASE-INSENSITIVE

while( __ICASE(*name1)==__ICASE(*name2) )

This is no coincidence. That bug (and others like bad fseek() and absence of frename()) is what prompted me to write my own file system driver in the first place, so the solution was put in place for compatibility back in the hpgcc days.
Now this is legacy code, which means I didn't upgrade it to Unicode. It will only change 'a' to 'z', but doesn't do proper Unicode case-folding. The utf library can do it, but I thought it was overkill, especially since the file system will be used 99.9% of the time in case sensitive mode.

(10-18-2016 11:12 PM)matthiaspaul Wrote: I would like to suggest to extend this idea to the four special cases utilizing the two "case"-bits in the SFN discussed previously:

This can be done, but it's hard to find a use case where it actually helps.
I'd think that none of those bad implementations are case-sensitive, therefore any SFN written by those OS will be uppercase and there's nothing newRPL can do about it. On the other hand, a file written by newRPL would display the incorrect case if the OS doesn't support it, but the OS would be able to open the file normally, as the comparison would be case-insensitive (as you said, merely a cosmetic problem).
If such OS changes the directory entry it should preserve those bits, even if they don't support their behavior (otherwise it's a VERY bad implementation). If the file gets renamed, and the OS writes a completely new name, then the name will lose its case, but there's no way to solve this from newRPL. Even if newRPL writes a LFN, the OS might decide (during rename) that because the new name fits in 8.3 it doesn't get an LFN when it creates the new entry, and loses its case anyway.

Regarding the SFN fallback: I think if there's no room to create the entry, then file creation needs to fail and return an error (as-implemented). Creating a file with a name different from the requested one won't help the user find his file... not to mention that if there's no room for the directory entry the disk might be full. It's better to let the user know with an error, don't you think?

matthiaspaul · 10-19-2016, 11:25 PM

(10-19-2016 03:01 AM)Claudio L. Wrote: Regarding the SFN fallback: I think if there's no room to create the entry, then file creation needs to fail and return an error (as-implemented). Creating a file with a name different from the requested one won't help the user find his file... not to mention that if there's no room for the directory entry the disk might be full. It's better to let the user know with an error, don't you think?

Running out of free directory entries to be allocated for a LFN does not necessarily mean that a file cannot be created nor that the case of the filename cannot be preserved.
The size of the directory table for the root directory on FAT12 and FAT16 partitions is fixed - if the file would be created in the root directory, the filesystem may still have plenty of free clusters, but still run out of directory entries. In fact, this is a rather common problem when using VFAT LFNs on FAT12 and FAT16 volumes, as the root directory allows for a maximum of 512 directory entries on many systems. (Of course, the problem does not exist in sub-directories.)
If there is no free directory entry for the LFN and the file's filename and extension are all-uppercase or all-lowercase, it is still possible to preserve the case via the "case"-bit method on systems supporting this extension - so, it's not that the case could not be stored (which should cause an error message), more that not all platforms will be able to retrieve this info later on (not sure, if this already warrants bailing out with an error message prophylactically).
In the other cases, there should be at least one cluster which could be used for the file - not much, but it may be sufficient for the file.
So, always bailing out with an error message as soon as there are no more free directory entries to be allocated for a LFN appears to be a bit premature to me - however, I totally agree, that the system should issue an error when there are no more directory entries for a LFN if the case of the given filename is mixed and therefore cannot be stored using the "case"-bit method (unless the system is in case-insensitive mode).

Greetings,

Matthias

matthiaspaul · (This post was last modified: 10-20-2016 09:17 AM by matthiaspaul.)

There's another compatibility enhancement which could be implemented into newRPL to automatically work around a bug in the original HP 49g+/50g firmware.

As we have seen in the other thread, when the original firmware creates LFNs, it does not perform any translation of the HP49/50 character set into Unicode, codepoints 80h..FFh are just zero-padded to 16-bit to fit into the UCS-2 slots.
This is fine for codepoints A1h..FFh, which are the same for the HP49/50 character set, ISO 8859-1 and Unicode. However, the code assignments differ for codepoints 80h..A0h.

While codepoints 80h..A0h are valid characters in LFNs, at least 80h..9Fh do not make any sense at all in a filename (they are control codes in Unicode). This could be used for an heuristical extension:

If newRPL encounters a code 0080h..009Fh in a LFN, it could blindly map these invalid codes to the correct Unicode codes:

0080h (PAD) -> 2221h (∡)
0081h (HOP) -> 0101h (ā)
0082h (BPH) -> 2207h (∇)
0083h (NBH) -> 221Ah (√)
0084h (IND) -> 222Bh (∫)
0085h (NEL) -> 03A3h (Σ)
0086h (SSA) -> 25B6h (▶)
0087h (ESA) -> 03C0h (π)
0088h (HTS) -> 2202h (π)
0089h (HTJ) -> 2264h (≤)
008Ah (LTS) -> 2265h (≥)
008Bh (PLD) -> 2260h (≠)
008Ch (PLU) -> 03B1h (α)
008Dh (RI) -> 2192h (→)
008Eh (SS2) -> 2190h (←)
008Fh (SS3) -> 2193h (↓)
0090h (DCS) -> 2191h (↑)
0091h (PU1) -> 03B3h (γ)
0092h (PU2) -> 03B4h (δ)
0093h (STS) -> 03B5h (ε)
0094h (CCH) -> 03B7h (η)
0095h (MW) -> 03B8h (θ)
0096h (SPA) -> 03BBh (λ)
0097h (EPA) -> 03C1h (ρ)
0098h (SOS) -> 03C3h (σ)
0099h (SGCI) -> 03C4h (τ)
009Ah (SCI) -> 03C9h (ω)
009Bh (CSI) -> 0394h (Δ)
009Ch (ST) -> 03A0h (Π)
009Dh (OSC) -> 03A9h (Ω)
009Eh (PM) -> 25A0h (■)
009Fh (APC) -> 221Eh (∞)

00A0h (NBSP) -> 20ACh (€)

This doesn't work for 00A0h, as NBSP cannot be ruled out as a valid character in a filename. So, without further info newRPL cannot decide if 00A0h means 20ACh (as for files created under the original HP 49g+/50g firmware) or 00A0h (as for files created anywhere else, including newRPL).
One such further indicator could be the following: If at least one LFN character is from the 0080h..009Fh range, the calculator knows that the file was created on the HP49/50 and thus can reliably assume that 00A0h means 20ACh. Obviously, this automatic detection does not work for filenames without any of these characters, but as it does not cause any harm, it is better than nothing already. Without this indicator, it is more safe to assume 00A0h, and translating this to 20ACh might require a special "compatibility" mode to be activated by the user.

In either case, this extension only works for the direction "file created under original firmware and read under newRPL". Translating the correct Unicode characters back into the 0080h..009Fh range, so that they are understood by the original firmware, should happen only in a special "compatibility" mode (if at all).

Greetings,

Matthias

Claudio L. · 10-20-2016, 03:01 AM

(10-19-2016 11:25 PM)matthiaspaul Wrote: ...it is still possible to preserve the case via the "case"-bit method on systems supporting this extension...

Oh, I understand now. You were assuming newRPL with the modifications you suggested, which potentially use unnecessary LFN entries. I was thinking as it is now, where it uses the least number of entries needed to guarantee the Unicode name preservation (therefore losing the LFN would give you a SFN potentially with a ~nn tail, good luck finding your file later).
It makes sense now.

Claudio L. · 10-21-2016, 01:21 PM

(10-16-2016 09:53 PM)LinusSch Wrote: I think I barely qualify as a beginner level programmer, but one of the languages I can make use of is C++. Point me to where I can help out! (but expect extremely slow progress)

I have another idea where you could help. I noticed in another thread you were thinking of adding a skin to the simulator. Right now the simulator is a plain Qt application. I have a skeleton for a Qt Quick application with the skin, resizeable screen, etc. but it doesn't have the "guts" of the newRPL core ported to it, so all it does is look good.
Would you prefer to work on something like that instead of the newRPL core? Several forum members expressed in the past they would like a skinnable simulator.
Let me know, the skinnable code is not in repository yet, as it was just my own experiment for the time being, but I can give you the code as a starting point.

Claudio

LinusSch · 11-05-2016, 05:12 PM

I haven't worked with sourceforge, GitHub or even git itself yet so if anyone can point me to a good primer on that I would very much appreciate it.

And I don't know Qt, but it seems like a good project to try. I am very much interested in creating stuff that is pleasing to use and your coding priorities seem to align well with mine. :) Will you put it in the repository, or should I pm you my email?

Maybe I should note that I feel quite comfortable with C, even though I have only really worked in related languages - not the ancestor itself. And I have no experience at all with more than some basic macros in RPL.

When you say to read all documentation at hpgcc.org, I guess you mean all the newRPL documentation - no need to read up on the hpgcc project. Correct?

Looking forward to trying to contribute! :) /Linus

Claudio L. · 11-05-2016, 06:53 PM

(11-05-2016 05:12 PM)LinusSch Wrote: I haven't worked with sourceforge, GitHub or even git itself yet so if anyone can point me to a good primer on that I would very much appreciate it.

And I don't know Qt, but it seems like a good project to try. I am very much interested in creating stuff that is pleasing to use and your coding priorities seem to align well with mine. Will you put it in the repository, or should I pm you my email?

Maybe I should note that I feel quite comfortable with C, even though I have only really worked in related languages - not the ancestor itself. And I have no experience at all with more than some basic macros in RPL.

When you say to read all documentation at hpgcc.org, I guess you mean all the newRPL documentation - no need to read up on the hpgcc project. Correct?

Looking forward to trying to contribute! /Linus

I meant only newRPL docs, no need for the rest. I don't have access to that code right now (it's on another machine), I'll commit it to the repos in a few days and let you know.
Regarding git docs, just google for tutorials on Git and that's all you need. On sourceforge you can go to the project and create your own fork, then work on your own repository and whenever you have something ready for prime time you send a merge request.

LinusSch · 11-06-2016, 11:18 PM

Sounds good, here I go... Smile

Claudio L. · 11-08-2016, 05:31 PM

November update!

The new ROM has many changes. I recommend to decompile your programs, store them as text before updating, then compile again from source after the update. A full memory clear shouldn't be necessary.

Changes in this version:
* Menus are enabled! Finally, although a couple of libraries still don't have menus, most menus are in place.
* Main menu assigned to SYMB key. Other menus: ARITH, CMPLX (key 1), TIME (key 9), PRG (key N) are also available from the keyboard (also from the main menu). The CHARS (key N) is VAR for the time being.
* On-VAR swaps the menus
* On-VAR long press hides the second menu
* Added many commands for polynomials.
* Added Date and Time commands
* Some string manipulations commands discussed in this forum were added.

Please play with all the menus, review and comment on the organization. Also please check the help for each command and report problems (doesn't fit on screen, not clear enough, etc).

Han · 11-08-2016, 06:20 PM

Can't wait to get home to try this out!

Claudio L. · 11-08-2016, 08:28 PM

(11-08-2016 05:31 PM)Claudio L. Wrote: Also please check the help for each command and report problems (doesn't fit on screen, not clear enough, etc).

Not sure if it was mentioned before: the help is long-press on each menu item (same as variable preview).

Han · (This post was last modified: 11-08-2016 09:01 PM by Han.)

One of the things I would like tinker with is some of the linear algebra commands that are currently not available. I just happened to notice a few hard-coded limits in the array size (65535). Is there any reason not to use define statements so that it could easily be managed later on rather than a hard value (I suspect that perhaps memory is the limiting factor)?

EDIT: Some initial questions:

The current code base appears to keep all the code for a particular command embedded within the library as suggested by:

Code:

void sample_handler()

{

  // If the current opcode is the prolog of an object...

  if(ISPROLOG(CurOpcode)) {

     // ... issue an error, since this example library only defines a command, not

     // objects with data payload

     // Raise a "Bad Opcode" exception

     Exception=EX_BADOPCODE;

     // Indicate which instruction caused the error (given by the RPL Instruction Pointer, IPtr)

     ExceptionPointer=IPtr;

     return;

  }

     switch(OPCODE(CurOpcode))

         {

          case MYCMD:

          ...

          // Also process all mandatory opcodes

          case ....

          ...

          default:

          // Raise a "Bad Opcode" exception

          Exception=EX_BADOPCODE;

          // Indicate which instruction caused the error (pointer to by the RPL Instruction Pointer, IPtr)

          ExceptionPointer=IPtr;

          return;

         }

}

I do not see a quick way to make use of existing routines (cross-library usage). For example, if I wanted to re-use the code of MYCMD, and since it is not defined as a function, what is the proper way to call MYCMD? Would I have to mix "SysRPL" like in

Code:

const WORD const matrixeval_seco[]={

    MKPROLOG(DOCOL,5),

    MKOPCODE(LIBRARY_NUMBER,DOMATPRE),     // PREPARE EACH ELEMENT

    (CMD_OVR_EVAL),    // DO THE EVAL

    MKOPCODE(LIBRARY_NUMBER,DOMATPOST),    // POST-PROCESS RESULTS AND CLOSE THE LOOP

    MKOPCODE(LIBRARY_NUMBER,DOMATERR),     // ERROR HANDLER

    CMD_SEMI

};

Thanks ahead of time for the help.

Guenter Schink · 11-08-2016, 10:56 PM

(11-08-2016 05:31 PM)Claudio L. Wrote: November update!

...

Are you sure the link at post#1 loads the new ROM? I don't see any difference to the previous one. No menu at all except "units"

Günter

Vtile · (This post was last modified: 11-09-2016 11:41 AM by Vtile.)

Finally I installed the Demo.. Still trying to figure out the key layout.

I also have one QoL improvement idea. Here is a square root of 3 as we see there is plenty of decimals.

Wouldn't it be good to see on the information area the running number of cursor to show which decimal you are looking at?
Edit. To palying with the idea even further: would there be any use of a functionality of rounding, splitting and copying the number from certain point to stack. I don't know if this would be handy to anything, atleast I don't see a personal use, but like said playing with the idea.

Claudio L. · 11-09-2016, 09:10 PM

(11-08-2016 10:56 PM)Guenter Schink Wrote:
(11-08-2016 05:31 PM)Claudio L. Wrote: November update!

...

Are you sure the link at post#1 loads the new ROM? I don't see any difference to the previous one. No menu at all except "units"

Günter

Oops, my apologies. I uploaded the file with a typo in the name, so it didn't overwrite the old one. Please download again now.

Claudio L. · 11-09-2016, 09:41 PM

(11-08-2016 08:45 PM)Han Wrote: One of the things I would like tinker with is some of the linear algebra commands that are currently not available. I just happened to notice a few hard-coded limits in the array size (65535). Is there any reason not to use define statements so that it could easily be managed later on rather than a hard value (I suspect that perhaps memory is the limiting factor)?

No particular reason, hard-coded constants could be replaced with named constants to beautify the code. In many cases it was, there might have been a few places where this was done in a rush or something.
Now the hard coded limit on arrays is to use 1 word for both width and height, 16 bits each. Raising that limit would imply redefining the object. I don't see any need for matrices larger than that in a calculator, but if you feel is too little we are still on time to redefine it.

(11-08-2016 08:45 PM)Han Wrote: EDIT: Some initial questions:

...

I do not see a quick way to make use of existing routines (cross-library usage). For example, if I wanted to re-use the code of MYCMD, and since it is not defined as a function, what is the proper way to call MYCMD?

There's 2 ways:
a) Move the code to a separate function and create an API in newrpl.h Since you've been looking at the matrix library, you can see its reusable code is in matrix.c, which can be used from other libraries. Some libraries have the functions in the same library .c file, others have a separate file.
b) If you are looking to execute a particular command, it's quite simple:
rplCallOperator() does it for you.

Code:

// COMPUTE THE SQUARE ROOT

rplCallOperator(CMD_SQRT);

if(Exceptions) {

   ...

}

There's 2 big limitations: One is that you can't use it for recursion, as the C stack is very limited (recursion must be done in RPL, using the return stack which can grow freely). The other is that you MUST make sure the command you call is monolithic C execution. In other words, you can't call any commands that rely on RPL code, as the RPL loop is not running.

To overcome this limitation you can use embedded RPL code that calls the command. This is the only safe way to execute commands like EVAL on a symbolic object for example. The symbolic could contain a user-defined function, which would be RCL'd and the program executed during evaluation. For this you need a fully running RPL loop.
To include RPL code there's also 2 ways:
a) The way you described, with an ugly list of macro declarations. This was used extensively on libraries, mostly because they were developed before newrpl-comp was available.
b) Add a .nrpl file to the project, and use the INCLUDE_ROMOBJECT(...) macro. This is much cleaner if you need RPL routines longer than 3 or 4 commands.

Claudio L. · (This post was last modified: 11-09-2016 09:57 PM by Claudio L..)

(11-09-2016 11:22 AM)Vtile Wrote: Finally I installed the Demo.. Still trying to figure out the key layout.

I also have one QoL improvement idea. Here is a square root of 3 as we see there is plenty of decimals.

Wouldn't it be good to see on the information area the running number of cursor to show which decimal you are looking at?
Edit. To palying with the idea even further: would there be any use of a functionality of rounding, splitting and copying the number from certain point to stack. I don't know if this would be handy to anything, atleast I don't see a personal use, but like said playing with the idea.

I'm not sure I understood what you mean. If you are talking about copy/paste portions of a number, you can already do that.
Use Left-Shift + left/right cursor to define a block, Left-Shift-Hold+Left to copy, Left-Shift-Hold+Right to paste.

EDIT: Actually, much easier way: just add a space wherever you want and hit enter, you'll have several numbers in the stack.

I could see use for an editor feature that would allow you to round a number at the cursor location, and delete all digits to the right.
Let's say you have 1.2345999999999... (32 digits total), so you edit the number, move the cursor to the right of the 5 and hit this special key, it will round the 5 to a 6 and delete the digits to the right without affecting the rest of the text in the editor.
Is that what you meant?

Vtile · (This post was last modified: 11-09-2016 11:43 PM by Vtile.)

Not actually that copy, paste, split and round functionality were more of thinking out loud. My primary improvement idea were cursors position counter that counts starting from radix. Just like you have cursor poition countr in most programming editors.

Written with anchor that believes to be smart phone.

Helix · 11-10-2016, 02:30 AM

(11-08-2016 05:31 PM)Claudio L. Wrote: Changes in this version:
* Menus are enabled! Finally, although a couple of libraries still don't have menus, most menus are in place.
* Main menu assigned to SYMB key. Other menus: ARITH, CMPLX (key 1), TIME (key 9), PRG (key N) are also available from the keyboard (also from the main menu). The CHARS (key N) is VAR for the time being.
* On-VAR swaps the menus
* On-VAR long press hides the second menu
* Added many commands for polynomials.
* Added Date and Time commands
* Some string manipulations commands discussed in this forum were added.

Here are some initial comments:

- At the beginning, the calculator displayed an "alarm event" message each time it was turned on. I think that the ACK command solved the problem.

- In the TIME menu, there are strange commands like MEMCHEK, MEMFIX, READCFI, PEEK, POKE, and also the MEM command that seems misplaced.

- The PRG / TEXT menu is identical to the PRG / LOCAL menu

- It would be nice if the help text could also display the entire name of the command, especially for long names that are partially masked in the soft menus.

- Currently the commands are displayed in the first soft menu (the black one). Is there a flag which would allow the commands to appear directly in the white soft menus?

- for the MENUSWAP shortcut, I don't like having to press too keys at the same time. I would prefer for example Right shift then Right cursor (I don't remember if this sequence is already used somewhere). On the other hand, I suppose I'll be able to make my own shortcuts when key assignments are implemented.

- the Right Shift CHARS shortcut is useless, since SYMB / VARS do the same thing, with two key presses as well.