List Commands Library for 50g

[DavidM]

Release Candidate 1 Posted

Testing for the ListExt library has gone well, and I've now been able to test all functions in a variety of settings on several different platforms. Specifically, the following emulated systems:

- 49G
- 48gii
- 49g+
- 50g

...and the following real hardware:

- 49g+
- 50g

In each case, the library was tested from all applicable ports and both in exact and approximate modes for all commands.

Several issues were discovered during testing (see the release notes for details). All were fixed in the "RC1" release attached to the first post in this thread. The command description documentation was also updated to reflect the current status of all commands.

I believe the library is now in a final state and "ready for prime time". In order to make it easier to test every command, I've included a special TESTALL command for this release on the second page of the main menu (you'll see it in the middle showing as "TESTA"). It would be wonderful if some of you would run that command on your systems and let me know the results. TESTALL checks each command in a variety of ways, including standard uses, edge cases, and any designated error traps. Verbose results are saved in a string stored in your HOME directory named "LETestResults". TESTALL by itself is larger than the combined code for the rest of the library, so it will of course be removed for the final release.

I'd be most grateful if someone who has access to a real 49G and/or 48gii could run TESTALL on those systems. While it runs successfully on emulated versions of them, I'd still like to hear from anyone that can verify it on real hardware. For a general frame of reference, TESTALL completes on my 50g (port0/approximate mode) in about 74 seconds. When run from port 1 in exact mode, it takes about 95 seconds.

[John Keith]

In Port 2 on a real 50g, about 95s in exact mode, 93s in real mode. All tests passed.

[DavidM]

Thanks John!

[DavidM]

The final version (1.1.2) of the ListExt library has now been posted to the General Software Library!

I was able to complete testing on a 49g successfully, which found no problems with the library on that platform. Interestingly I did run across a bug in the 1.19-6 ROM, though, specifically pertaining to the result of an exact integer list being multiplied by an exact integer constant -- which mysteriously gets converted to a real result. This is probably documented somewhere, though I don't recall seeing it. You can see this simply by executing "{ 2 } 3 *" on a 49g set to Exact mode with this ROM version. The result is a real { 6. } instead of integer { 6 } as it is on later ROM versions. The bug doesn't happen if you reverse the order of the arguments, so it's probably specific to the dispatch code handling in that ROM release.

In case anyone still tries to run TESTALL (if you still have the release candidate), there was a problem with it that occurs if you have the fraction mark set to comma (I know, I should have realized that one in advance). It will report failures of at least 8 tests in that scenario. The problem wasn't with the library, it was with TESTALL. I fixed the bug, but that command was removed from the final version so the fix will only apply to any future release.

Other than cosmetic changes to the About text (and removing TESTALL), the only change since the release candidate was a performance enhancement for LSEQ and LSEQR. The results of those commands are now built from scratch without using any of the built-in math functions, which cuts out the "middle man" in the main loop. As a result, they are quite zippy. Because there's more data to move with reals than there is with integers, real results are actually a bit slower than integers now. But you won't be waiting much on those commands to finish. 15000 LSEQ now completes in 1.93s (approximate mode) and 1.56s (exact mode). More down-to-earth list sequences finish much faster, of course.

Thanks again to everyone who participated with this project!

[Gilles59]

(10-31-2017 02:04 PM)DavidM Wrote:  The final version (1.1.2) of the ListExt library has now been posted to the General Software Library!

Bravo ! ;D

It would be nice to post ListExt on HPCALC.org too

[DavidM]

(10-31-2017 03:37 PM)Gilles59 Wrote:  Bravo ! ;D

It would be nice to post ListExt on HPCALC.org too

Thanks, Gilles. I’ve already submitted it there. Apparently there’s also an older version there that someone else already uploaded — I’ve asked Eric to replace the older one with the final release. I’m sure he will tak care of it when he gets a chance.

[John Keith]

Excellent! Congratulations on a job well done!

[Thomas Okken]

(10-04-2017 04:22 AM)Thomas Okken Wrote:  

(10-02-2017 08:58 AM)Werner Wrote:  eg. the early Random Number generator used for the 41c used
c=.211327, a=9821, m=1. (This is from memory, but I think I'm right ;-)

You're absolutely right, and I know that one from memory as well. :-D

That is also the RNG that I use in Free42. This thread got me thinking, though: maybe I should replace that RNG? Even if its properties are generally OK, the RNG in the real HP-42S generates numbers with 12 significant digits, while the one from the HP-41C manual generates numbers with only 6 significant digits.

And now I discover that the HP-42S RAN function behaves exactly like RAND in the 48G! Give it the same seed, and the same numbers pop out.

The behavior for seed 0 is different, but still, it looks like I've stumbled upon another way to make Free42 a bit truer to the original. Neat!

In Free42 2.0.7, I changed the implementation of the RAN and SEED functions according to https://groups.google.com/forum/m/#!msg/...tzMtZhlGoJ.

Apart from their names, the 42S RAN and SEED functions are identical to their RPL counterparts, RAND and RDZ, the only remaining difference being the starting seed after a cold start, which is 999500333083533 in the RPL calculators, and 2787 in the HP-42S.

While I'm pleased to have been able to implement RAN and SEED so that they behave exactly like their counterparts on the original calculator, there is one thing that bothers me, namely, the way RAN behaves near zero.

As was pointed out earlier in this thread, the RPL RAND function never returns zero (and neither does RAN on the 42S), which seems odd, given that the calculator's manual indicates that that functions returns numbers 0 <= x < 1 (apart from one place where the 48S AUR says 0 <= x <= 1, which is wrong but doesn't have anything to do with the behavior near zero). The 42S manual mentions the range of RAN twice, and states 0 <= x < 1 in both places.

The reason why the RPL RAND and the 42S RAN functions never return zero is because of the way they truncate the 15-digit internal number before returning it.

The RNG takes a 15-digit integer, multiplies it by 2851130928467 (a prime), takes the result of that multiplication modulo 10^15, keeps that as the next seed, and chops off the three least significant digits, and divides that result by 10^15, giving a uniformly distributed result that is >= 0 and < 1... except: when that final number is < 0.1, it only chops off the last two digits, and when it is < 0.01, it only chops off the last one digit, and when it is < 0.001, it doesn't chop off any digits... And because the least significant digit of the 15-digit seed is never zero (it is always 1, 3, 7, or 9), this means that the number that is returned is never zero.

This doesn't seem to make sense. The result of this algorithm is that RAN or RAND never return zero, which isn't necessary given that the documentation states that zero is a possible result, and it means that the distribution in the intervals [0, 0.001), [0.001, 0.01), and [0.01, 0.1) are all different, and don't match the distribution in [0.1, 1).

As long as you truncate the result from RAN or RAND to 12 digits after the decimal, this is all fine... but it seems to me like you shouldn't have to. I think this behavior is wrong. RAN and RAND should just chop off the last 3 digits of the 15-digit seed, divide the result by 10^15, and thus return a truly uniformly distributed number on the inverval [0, 1).

Other than this being a rare mathematical boo-boo by HP, is there any other explanation why these functions work the way they do?

[DavidM]

(11-21-2017 05:43 AM)Thomas Okken Wrote:  ...
As long as you truncate the result from RAN or RAND to 12 digits after the decimal, this is all fine... but it seems to me like you shouldn't have to. I think this behavior is wrong. RAN and RAND should just chop off the last 3 digits of the 15-digit seed, divide the result by 10^15, and thus return a truly uniformly distributed number on the inverval [0, 1).

Other than this being a rare mathematical boo-boo by HP, is there any other explanation why these functions work the way they do?

I've been hoping others would respond to the above, because the truncation method you advocate in the first paragraph above is exactly what I used for the LSHUF command in the library (it also does a range check, but that's a different issue altogether). I fear that the very people who are likely to have insight to this are no longer following this thread. Perhaps a separate posting of your observations might get more readers.

As to the "what were they thinking" issue, could it simply be that they were attempting to retain as many of the internal 15 digits generated as possible? I'm not qualified to comment on the efficacy of that approach in this instance, but it would at least be consistent with the general operation of most internal functions.

Regarding the exclusion of 0 from the possible results of the internal functions, I find it odd as well that the documentation is so out of sync with the implementation. The fact that the documentation is inconsistent implies that at least one person was looking at that somewhere along the lines, and you'd think that it would have been addressed properly as a result.

[Thomas Okken]

(11-22-2017 05:01 PM)DavidM Wrote:  

(11-21-2017 05:43 AM)Thomas Okken Wrote:  ...
As long as you truncate the result from RAN or RAND to 12 digits after the decimal, this is all fine... but it seems to me like you shouldn't have to. I think this behavior is wrong. RAN and RAND should just chop off the last 3 digits of the 15-digit seed, divide the result by 10^15, and thus return a truly uniformly distributed number on the inverval [0, 1).

Other than this being a rare mathematical boo-boo by HP, is there any other explanation why these functions work the way they do?

I've been hoping others would respond to the above, because the truncation method you advocate in the first paragraph above is exactly what I used for the LSHUF command in the library (it also does a range check, but that's a different issue altogether). I fear that the very people who are likely to have insight to this are no longer following this thread. Perhaps a separate posting of your observations might get more readers.

Good idea. This discussion is off-topic for this thread. Here's a new one.

[StephenG1CMZ]

(08-13-2017 11:06 PM)DavidM Wrote:  Re: LSPLIT<whatever>

"LSPLT" fills up a menu. "LSPLIT" shows as "LSPLI", as would "LSPLITR". So if both of those commands are to be included, you would see LSPLI LSPLI in the menus, but the entire commands would show as normal when keying in a program. Is this a problem? I could see it being a bit confusing for someone who wasn't already familiar with both commands. Then again, I'm not sure how useful any of this is without the documentation anyway. I find myself referring to the documentation more and more now, and I should know the commands better than anyone!

Re: SORT helper

I could see having a specific "wrapper" command that would take two lists (L1 in SL1 and L2 in SL2) and do the following:

- check to make sure both lists have the same element count
- combine them as follows:
- { { L1E1 { L2E1 } } { L1E2 { L2E2 } } ... { L1EN { L2EN } } }
- perform a sort (I can probably check for the existence of Werner's LSORT and use that if present)
- split the lists apart again

I believe forcing them into that structure should protect the objects adequately.

So then the question becomes what to name it. LSORT2 (or LSORT<anything>) would show on the menu as LSORT, which then might be easily confused with a simple sort command. And since a person might easily have 2 lists on the stack anyway, this could be a little riskier than usual -- you might not even realize that you just sorted 2 lists instead of 1 until it's too late. I would probably want to make the two stand out -- call it something like L2SRT (or L2SORT).

Thoughts?

As I understand the naming problem, SORT and SORT2 cannot be distinguished in a 4-character menu. But SORT2 is desirable because it is more memorable than S2RT.

Would it be practical to have two versions of the names (calling the same routine for implementation)?

S1RT //Selectable from menu
S2RT //Selectable from menu
LONGSORT1 //memorable and descriptive name
LONGSORT2 //memorable and descriptive name
(The LONG prefix avoids seeing SORT and assuming SORT1).

[Not sure how useful that suggestion would be: The Prime has 31-char names]

[DavidM]

(12-30-2017 08:59 PM)StephenG1CMZ Wrote:  As I understand the naming problem, SORT and SORT2 cannot be distinguished in a 4-character menu. But SORT2 is desirable because it is more memorable than S2RT.

Would it be practical to have two versions of the names (calling the same routine for implementation)?

S1RT //Selectable from menu
S2RT //Selectable from menu
LONGSORT1 //memorable and descriptive name
LONGSORT2 //memorable and descriptive name
(The LONG prefix avoids seeing SORT and assuming SORT1).

[Not sure how useful that suggestion would be: The Prime has 31-char names]

Menu items on the 50g contain up to the first 5 characters of the ID, and in this particular case we ended up with KSORT for the name to designate that it was a "Key SORT". "KSORT" seemed OK to stand on its own, so no aliases were defined for that particular command.

That could always be changed if needed in some future release, of course. :-)

The actual names for commands and IDs can definitely be longer than 5 chars. I know you can create globals with names up to 127 chars, but I believe library command names are limited to 16 chars.

[Gilles59]

+Hi David, suggestion for a future release ?

Perhaps I miss something but I dont see an INSERT command in the Library (the contrary of LRMOV) . It could be LINSR

{ "a" "b" "c" "e" }
"d"
4
LINSR

-> { "a" "b" "c" "d" "e" }

and

{ "a" "b" "c" "e" }
{ "x" "y"}
4
LINSR
-> {"a" "b" "c" "x" "y" "e" }

[DavidM]

(01-06-2018 03:49 PM)Gilles59 Wrote:  +Hi David, suggestion for a future release ?

Perhaps I miss something but I dont see an INSERT command in the Library (the contrary of LRMOV) . It could be LINSR

{ "a" "b" "c" "e" }
"d"
4
LINSR

-> { "a" "b" "c" "d" "e" }

and

{ "a" "b" "c" "e" }
{ "x" "y"}
4
LINSR
-> {"a" "b" "c" "x" "y" "e" }

Hmmm.... that's an amazingly intuitive command to have been missed by so many people (me included ). I was thinking that's what GoferList's Insert did, but after looking at it more closely, Insert is more like an "append". That's an odd misnomer, given that "+" is much easier to enter into a program (and it essentially does the same thing as far as I can see). Insert does at least call AppendList, which might perform better than >TCOMP in some cases (which is what + does).

I'll add that to the list of future enhancements.

[brickviking]

(01-06-2018 07:29 PM)DavidM Wrote:  

(01-06-2018 03:49 PM)Gilles59 Wrote:  +Hi David, suggestion for a future release ?

Perhaps I miss something but I dont see an INSERT command in the Library (the contrary of LRMOV) . It could be LINSR

{ "a" "b" "c" "e" }
"d"
4
LINSR

-> { "a" "b" "c" "d" "e" }

and

{ "a" "b" "c" "e" }
{ "x" "y"}
4
LINSR
-> {"a" "b" "c" "x" "y" "e" }

Hmmm.... that's an amazingly intuitive command to have been missed by so many people (me included ). I was thinking that's what GoferList's Insert did, but after looking at it more closely, Insert is more like an "append". That's an odd misnomer, given that "+" is much easier to enter into a program (and it essentially does the same thing as far as I can see). Insert does at least call AppendList, which might perform better than >TCOMP in some cases (which is what + does).

I'll add that to the list of future enhancements.

Is there much difference between this method and {newarray} {oldarray} + if you were doing {oldarray} {newarray} 0 LINSR ? Just something I thought of.

(Post 149)

[pier4r]

(01-06-2018 03:49 PM)Gilles59 Wrote:  { "a" "b" "c" "e" }
{ "x" "y"}
4
LINSR
-> {"a" "b" "c" "x" "y" "e" }

Nice point gilles! Impressive that no one needed it so far

Also small but potentially difficult (for later later releases) addendum
{ "a" "b" "c" "e" }
{ "x" "y"}
{ 4 4} //this can be a though decision, but one can fix a convention nd then that's it.
LINSR
-> {"a" "b" "c" "y" "x" "e" }


{ "a" "b" "c" "e" }
{ "x" "y"}
{ 1 3 } //I want to insert x where 'a' is and push 'a' and 'y' where 'c' is and push 'c'
LINSR
-> {"x" "a" "b" "y" "c" "e" }

[DavidM]

(01-06-2018 10:55 PM)brickviking Wrote:  Is there much difference between this method and {newarray} {oldarray} + if you were doing {oldarray} {newarray} 0 LINSR ? Just something I thought of.

Concatenating lists is easy enough with + (or in SysRPL AppendList). I believe the more salient part of what Gilles is requesting is the insertion of an object (or list of objects) in between pre-existing ones in a list.

While he didn't specify this, I would assume that an index parameter ≤ 1 would result in the new object/s being placed at the beginning of the target list, and an index > the list length would place the object/s at the end. Any other index would insert the objects beginning at the position indicated.

Does that make sense?

[DavidM]

(01-06-2018 11:15 PM)pier4r Wrote:  Also small but potentially difficult (for later later releases) addendum
{ "a" "b" "c" "e" }
{ "x" "y"}
{ 4 4} //this can be a though decision, but one can fix a convention nd then that's it.
LINSR
-> {"a" "b" "c" "y" "x" "e" }


{ "a" "b" "c" "e" }
{ "x" "y"}
{ 1 3 } //I want to insert x where 'a' is and push 'a' and 'y' where 'c' is and push 'c'
LINSR
-> {"x" "a" "b" "y" "c" "e" }

Are you just trying to make my head spin?

The targeted positions would incrementally be off by one for each step, which would lead to some pretty crazy logic in a program. It's difficult for me to see how this would be anything but confusing if trying to troubleshoot a program where it was used this way. I'd be more inclined to put placeholders in a list and then use PUT (or PUTI) to replace the placeholders as needed. At least that way there would be a direct correlation to the indices with the final positions in the list.

How would you use this kind of construct? I'll admit it may simply be my lack of imagination for understanding how useful this could be.

[pier4r]

(01-07-2018 01:47 AM)DavidM Wrote:  Are you just trying to make my head spin?

Did you discover my real objective only now?

Yes it is difficult because the list changes after every addition, so I understand clearly that one may say "look, if you want multiple inserts, use this command multiple times". First simple but useful steps.

For the multiple inserts at once. I do not have a real use case, I only thought that there can be actions done a single time or multiple times (the idea is list processing at once, isn't it?).

Like position, or multiple positions.
Removal or multiple removal.
Swapping or multiple swaps.
Insertion or multiple insertions.
Replacement or multiple replacements.
etc...

I am not sure whether those may be used only in remote edge cases though. And yes one has to stick to a convention, like "the index that are defined for multiple insertion will be used as-is" so at the end is the user that has to care about how it will work.

Like

{ a b c d e } { f g } { 4 4 } -> { a b c g f e}
(sort of stack push)

{ a b c d e } { f g } { 2 4 } -> { a f b g c e}
{ a b c d e } { f g } { 4 2 } -> { a g b c f e}

[DavidM]

(01-07-2018 12:43 PM)pier4r Wrote:  For the multiple inserts at once. I do not have a real use case, I only thought that there can be actions done a single time or multiple times (the idea is list processing at once, isn't it?).

After thinking about this a bit more, I believe the best course of action is simply to create the command in the more simple form with only two overloaded argument configurations:

3: { original list }
2: <object>
1: <target position>

or

3: { original list }
2: { <object1> <object2> ... <objectN> }
1: <target position>

Achieving Pier's alternative form should be easy enough with a DOLIST command:

5: { original list }
4: { <object1> <object2> ... <objectN> }
3: { <target1> <target2> ... <targetN> }
2: 2
1: << LINSR >>
DOLIST

Using large lists with that final construct would be a good torture test. Copious amounts of temporary objects would be created and moved around in the process.