List Commands Library for 50g

[snrowe]

(09-24-2017 03:37 PM)DavidM Wrote:  Not in the way that I think you mean. Part of this stems from the fact that the actual memory footprint of a series of discrete objects in TEMP is not the same as the memory layout of a list containing those same objects in TEMP. So there's not a direct way to simply "remove the list prologue and SEMI" and put pointers to all the elements on the stack.

Yes I think what you're describing is what I was hoping for. Sounds like it's not as straight forward as I had hoped.

(09-24-2017 03:37 PM)DavidM Wrote:  I suppose it may be possible using Saturn code to isolate a given string within a list in TEMP, remove the surrounding (extraneous) list structure, compress the TEMP slot, move all slots above it to keep the chain intact, etc., but that would be a massive undertaking.

Yes was thinking there must be a way just didn't know how feasible. Guess if it was easy someone would have done it already

(09-24-2017 03:37 PM)DavidM Wrote:  One other thought I've had for this: have you considered storing the string to port memory, dropping references, forcing a GC, then recalling/purging? If you have room in one of the ports, that might be a workable alternative.

My main working program is on the HP48 SX which I have the program in a protected port 2. The main program is at 80k and supporting libraries take up the rest of the port. I use port 1 as merged memory which gives me a reasonable amount of ram. So saving to a port will usually not help much from a memory aspect in this case.

On the version I'm making for the hp50 I may save data to the port as there is more memory do so. On a separate topic I'll probably create a separate post for I'm having program speed issues when running my program as a library on the hp48G, hp49G or hp50G. I think it's related to the forced GC when I run the MEM command from a library opposed to a program stored as a variable.

Thanks for the feedback and hope your able to find a workaround for your list project the list functions look very useful.

[snrowe]

(09-24-2017 04:13 PM)DavidM Wrote:  

(09-24-2017 03:37 PM)DavidM Wrote:  I suppose it may be possible using Saturn code to isolate a given string within a list in TEMP, remove the surrounding (extraneous) list structure, compress the TEMP slot, move all slots above it to keep the chain intact, etc., but that would be a massive undertaking.

Actually, this may not be as far-fetched as I thought. The Saturn routine Shrink$Any might be very useful for something like this. I don't have any direct experience with using that type of approach, but perhaps others here might have more insight.

While it would only help with a single list element, I could see an approach that does the following:

- MOVEDOWN the string into the starting position of the list (wiping out the list prologue)
- Adjust the string length field to encompass the entire list size (including the SEMI at the end)
- Shrink$Any the string to the original length, which may in fact handle the rest of the "massive undertaking" that I mentioned above.

Anyone know enough about Shrink$Any to know if this is a sensible approach?

Nice! Look forward to seeing what you and others think about your approach above.

[DavidM]

(09-24-2017 05:41 PM)snrowe Wrote:  

(09-24-2017 04:13 PM)DavidM Wrote:  Actually, this may not be as far-fetched as I thought. The Saturn routine Shrink$Any might be very useful for something like this. I don't have any direct experience with using that type of approach, but perhaps others here might have more insight.

While it would only help with a single list element, I could see an approach that does the following:

- MOVEDOWN the string into the starting position of the list (wiping out the list prologue)
- Adjust the string length field to encompass the entire list size (including the SEMI at the end)
- Shrink$Any the string to the original length, which may in fact handle the rest of the "massive undertaking" that I mentioned above.

Anyone know enough about Shrink$Any to know if this is a sensible approach?

Nice! Look forward to seeing what you and others think about your approach above.

For some reason I was thinking that you were targeting a 49g and/or 50g for this app, both of which have Shrink$Any. The 48sx doesn't, which would complicate things (Shrink$ exists, but I believe it only works on the top-most item in TEMP -- which I believe is why Shrink$Any was created). If the large list is the last object created in TEMP, though, this might still work.

I may experiment with the concept when I get a chance, but it may be a while before I can get to it.

[Software49g]

Hello,

while this is principally possible you will soon notice, that this will result in a lot of overhead which unnecessary slows down of what your are wanting to achieve.

It is a lot faster (and easier to program) to simply allocate the needed memory and than copy the wanted data there.

So basically use CREATETEMP / GETTEMP / MAKE$ / MAKE$N which fit your needs best and then use MOVEDOWN or MOVEUP to copy your data from the source into the allocated memory.


Ensure that you create a valid object – never ever forget the prolog (and epilogue in case of composite objects) as all objects in TEMPOB memory muse be valid. Otherwise you will run in a crash sooner or later, you have been warned.


Also note, that ARM replacement version of original Saturn routines may behave different than what is given in the documentation for the Saturn routines.
For example:
MOVEDOWN in Emu48 behaves exactly as documented
whereas
MOVEDOWN in the Saturnator aka 49G+/50g does not update C.A so C.A is not equal to D1 at the end of the routine.
This applies also to SKIPOB and most likely there are other routines which do behave differently.


So if your working routine in Emu48 is crashing in the real 50g consider checking the appropriate registers.
GOSBVL =DBUG.TOUCHE might be of help for this.


HTH,
Andreas

[DavidM]

(09-25-2017 06:18 PM)Software49g Wrote:  ...It is a lot faster (and easier to program) to simply allocate the needed memory and than copy the wanted data there.

So basically use CREATETEMP / GETTEMP / MAKE$ / MAKE$N which fit your needs best and then use MOVEDOWN or MOVEUP to copy your data from the source into the allocated memory...

Good to see that you're following this discussion, Andreas! I know this is an area where you've had lots of experience.

I think the problem with the above approach in Scott's situation is that he doesn't have enough available memory to make the copy. In other words, the list element in question is larger than the free memory available. This is why I was suggesting the possibility of isolating that element by clearing out the surrounding list structure and resizing the object in TEMP. @Scott, please correct me if I've misunderstood this issue.

Of course this gets even messier if the list wasn't the last object in TEMP on his 48sx, since there would then be a need to move the higher TEMP slots in addition to the return stack after adjusting the target slot. I believe that Shrink$Any already handles that, but that functionality would have to be replicated for a 48sx.

[pier4r]

Request: given an ordered list (integer and/or reals), asearch function that returns the index of the closer value compared to the input.

Inspired by: http://www.hpmuseum.org/forum/thread-915...l#pid80283

[DavidM]

(09-26-2017 01:10 PM)pier4r Wrote:  Request: given an ordered list (integer and/or reals), asearch function that returns the index of the closer value compared to the input.

Wouldn't GoferList's FindIndex work for this?

[pier4r]

Hmm, did not check. I'll check

[John Keith]

(09-26-2017 03:29 PM)DavidM Wrote:  

(09-26-2017 01:10 PM)pier4r Wrote:  Request: given an ordered list (integer and/or reals), asearch function that returns the index of the closer value compared to the input.

Wouldn't GoferList's FindIndex work for this?

I can't see how. You can use Find or FindIndex to find the first element in the list that is within a certain amount of the desired value. Assuming the value you are searching for is in the variable n, you can do something like this:

Code:

\<< n - ABS .01 < \>> FindIndex

which will find the index of the first element that is within .01 of the value of n, or 0. if no element meets that criterion.

To find the closest value you can use the ListExt command KSORT like so:

Code:

DUP n - ABS KSORT HEAD SWAP HEAD

which will leave the closest element on level one and the error on level 2.

With regard to the thread that Pier is referring to, if the above code snippet is used with LSORT as the third argument to KSORT, it should be significantly faster than a binary search routine written in User RPL.

John

[pier4r]

Uhhhhhh, I missed KSORT.

That's super neat. Can I ask all the family of sorting for lists of two elements? I will formalize the request later.

We are on another level now. That's great. I will have to improve my algorithms and attack some new problems.

[DavidM]

(09-27-2017 11:47 AM)John Keith Wrote:  

(09-26-2017 03:29 PM)DavidM Wrote:  Wouldn't GoferList's FindIndex work for this?

I can't see how. You can use Find or FindIndex to find the first element in the list that is within a certain amount of the desired value. Assuming the value you are searching for is in the variable n, you can do something like this:

Code:

\<< n - ABS .01 < \>> FindIndex

which will find the index of the first element that is within .01 of the value of n, or 0. if no element meets that criterion.

To find the closest value you can use the ListExt command KSORT like so:

Code:

DUP n - ABS KSORT HEAD SWAP HEAD

I was doing a mental translation of Pier's request into what I thought he was trying to do in the other thread, which was looking for the largest number in the list which was still smaller than the target. I believe that is still doable with Find/FindIndex (you may need to reverse the list first, though).

John's solution is more accurate to the actual request Pier made, though.

KSORT is really just a "helper command" that combines two lists into a "list of lists" for either the built-in SORT or a program of your own choosing, then returns the lists to their original (but newly sorted) grouping. I haven't tried to reinvent the code for sorting, especially since Werner already did such a wonderful job with LSORT.

[DavidM]

As promised, I've updated the first post in this thread with the latest release (1.1.1d).

There's two new commands (LPOPR and LSSR), several performance enhancements (LSHUF, LSEQ/LSEQR), and a lot of housekeeping for this version. LPICK and LRMOV may be faster now when processing large lists due to their saving the list temporarily in the hidden directory. See the release notes for more detail.

LSHUF also uses the "save to hidden directory first" methodology, but in addition to that more of the code was moved to a Saturn code object to speed things up.

Many commands were updated to round numbers (as opposed to truncating) when an implied integer is needed for an argument. This is now the default mode of operation for all commands in the library.

If you happen to see that any of your favorite commands have been updated, please check them out to see if anything behaves differently than expected for you. Other than rounding vs. truncation, the housekeeping I've done shouldn't have caused any of the commands to change how they operate.

Thanks to everyone for your feedback and ongoing ideas and contributions! I'm hopeful that this is a very-nearly-finished version of the library.

[snrowe]

(09-25-2017 07:41 PM)DavidM Wrote:  

(09-25-2017 06:18 PM)Software49g Wrote:  ...It is a lot faster (and easier to program) to simply allocate the needed memory and than copy the wanted data there.

So basically use CREATETEMP / GETTEMP / MAKE$ / MAKE$N which fit your needs best and then use MOVEDOWN or MOVEUP to copy your data from the source into the allocated memory...

Good to see that you're following this discussion, Andreas! I know this is an area where you've had lots of experience.

I think the problem with the above approach in Scott's situation is that he doesn't have enough available memory to make the copy. In other words, the list element in question is larger than the free memory available. This is why I was suggesting the possibility of isolating that element by clearing out the surrounding list structure and resizing the object in TEMP. @Scott, please correct me if I've misunderstood this issue.

Of course this gets even messier if the list wasn't the last object in TEMP on his 48sx, since there would then be a need to move the higher TEMP slots in addition to the return stack after adjusting the target slot. I believe that Shrink$Any already handles that, but that functionality would have to be replicated for a 48sx.

Hi David, Yes you got my issue correct. I am porting the program to all calc platforms 48G, 49G, 49G+, 50G. I have a working old version on my 49G and 50G which I could use your method. But I may have solved my issues by not storing the strings in a list and instead storing them as variables in a directory. It makes it a bit slower for some things but allows me to work in low memory conditions. For the 48SX this seems to work satisfactory for what I'm doing, but if your shrinkany method works for the 49,50G I may try that out.

Thanks,
Scott

[pier4r]

(09-28-2017 02:08 AM)DavidM Wrote:  As promised, I've updated the first post in this thread with the latest release (1.1.1d).

Nice!

I'll try to use those commands that I overlooked, especially sorting/picking elements. The problem is to find challenges for them, not too overwhelming (like the trees as list, that I still have to do) and not too small. I guess I can play more in the direction of the thread with resistors to see what can I do.

(The other one is to play with statistics to simulate votes, but the memory of the 50g is not enough).

[DavidM]

I was going to post this as a poll, but quickly realized that the only way to create a poll is to start a new thread. I'd rather keep it in this thread, so please respond with your "vote" as appropriate.

The Issue

LSPLT and LRSPL are used to split a list into two parts. In its current form, the result is presented as a list with the two parts embedded as sublists. I've received a suggestion that a preferred approach might be to return the result simply as two separate lists instead of one, as shown below.

Current Example (Method 1)
{ 1 2 3 4 5 }
2
LSPLT

results:
1: { { 1 2 } { 3 4 5 } }

Alternative Example (Method 2a)
{ 1 2 3 4 5 }
2
LSPLT

results:
2: { 1 2 }
1: { 3 4 5 }

Alternative Example (Method 2b)
{ 1 2 3 4 5 }
2
LSPLT

results:
2: { 3 4 5 }
1: { 1 2 }

Please respond with your preferred approach, which I will assume applies to both LSPLT and LRSPL.

I don't have strong feelings about any of the above methods; I see merits to all of them. As such, I will either keep or change those commands for the next release based on the feedback received. I do think it's important that both LSPLT and LRSPL use the same approach, though (whatever it ends up being).

[pier4r]

2b.

For me I would choose actually 1 but for a practical perspective a lot of people do Stack acrobatics so the 2b would be the most reasonable:
L2: second split
L1: first split



Aside from that, I did not yet test KSORT and other overlooked commands but I am going to request commands to handle a sort of "associative" list.

Given a list of sublists that follows a certain format (when not, it is the user fault, the program has not to check the entire input), in particular:
{ {k1 v1} {k2 v2} ... {kn vn}}

Where k1 is the key and v1 is the value (the position may be also inverted, the important part is that it is well defined in the documentation).

Could be possible to have commands that sorts this list by value or by key?
A command that returns if a key exists (and how many times)?
A command that returns if a value exists (and how many times)?

Of course those are heavily nice to have commands, but since the library is already so beautiful, I ask.

edit: I am already happy if the above commands are done for keys as integers/string/reals and values only as reals.

[DavidM]

(09-29-2017 07:19 PM)pier4r Wrote:  2b.

Thanks for voting. You're the expert on how many votes are needed for this to be statistically meaningful!

(09-29-2017 07:19 PM)pier4r Wrote:  ...in particular:
{ {k1 v1} {k2 v2} ... {kn vn}}

Where k1 is the key and v1 is the value (the position may be also inverted, the important part is that it is well defined in the documentation).

Could be possible to have commands that sorts this list by value or by key?

The built-in SORT command already does this. It's the basis for how KSORT works -- it simply combines the two lists into this form, executes SORT (or your given program), then unbundles the combined lists back into their original (but now sorted) groups. Sorting two lists by the values would be done simply by swapping the value and key lists before executing KSORT.

(09-29-2017 07:19 PM)pier4r Wrote:  A command that returns if a key exists (and how many times)?
A command that returns if a value exists (and how many times)?

Of course those are heavily nice to have commands, but since the library is already so beautiful, I ask.

edit: I am already happy if the above commands are done for keys as integers/string/reals and values only as reals.

Wouldn't it be easier to simply keep the keys and values in their own lists? That way you achieve the above with { key or value list } <target> LCNT/MPOS/etc.

If you need to perform operations on the combined value/key lists in the above format, simply pass your program as the third argument to KSORT. Yes, I know the name is KSORT, but the supplied user program doesn't actually have to sort the list. See the command description for details. KSORT doesn't really care what the supplied program does, it just has to leave a list in the proper format on the stack when it's done so that the list can be unbundled properly.

Give it some thought. I think you'll find that keeping the keys and values as separate lists has some nice benefits, and KSORT provides some nice flexibility by allowing you to execute a user-supplied program on the bundled list.

[John Keith]

(09-29-2017 04:45 PM)DavidM Wrote:  I was going to post this as a poll, but quickly realized that the only way to create a poll is to start a new thread. I'd rather keep it in this thread, so please respond with your "vote" as appropriate.

The Issue

LSPLT and LRSPL are used to split a list into two parts. In its current form, the result is presented as a list with the two parts embedded as sublists. I've received a suggestion that a preferred approach might be to return the result simply as two separate lists instead of one, as shown below.

Current Example (Method 1)
{ 1 2 3 4 5 }
2
LSPLT

results:
1: { { 1 2 } { 3 4 5 } }

Alternative Example (Method 2a)
{ 1 2 3 4 5 }
2
LSPLT

results:
2: { 1 2 }
1: { 3 4 5 }

Alternative Example (Method 2b)
{ 1 2 3 4 5 }
2
LSPLT

results:
2: { 3 4 5 }
1: { 1 2 }

Please respond with your preferred approach, which I will assume applies to both LSPLT and LRSPL.

I don't have strong feelings about any of the above methods; I see merits to all of them. As such, I will either keep or change those commands for the next release based on the feedback received. I do think it's important that both LSPLT and LRSPL use the same approach, though (whatever it ends up being).

Not to be disagreeable, but I have to vote for 2a for several reasons:

Firstly, if the list that LSPLT returns is broken out onto the stack using OBJ\-> DROP the result is the same as 2a.

Second, if the two lists in version 2a are combined using + the result will be the original list and if they are combined using 2. \->LIST the result will be the list-of-lists that LSPLT currently returns.

Finally, version 2a is what GoferLists' Split command returns and I can see no need for the two commands to be opposite.

IMHO, version 2a would result in less stack manipulation than version 2b in most cases.

John

[snrowe]

(09-29-2017 11:30 PM)John Keith Wrote:  

(09-29-2017 04:45 PM)DavidM Wrote:  I was going to post this as a poll, but quickly realized that the only way to create a poll is to start a new thread. I'd rather keep it in this thread, so please respond with your "vote" as appropriate.

The Issue

LSPLT and LRSPL are used to split a list into two parts. In its current form, the result is presented as a list with the two parts embedded as sublists. I've received a suggestion that a preferred approach might be to return the result simply as two separate lists instead of one, as shown below.

Current Example (Method 1)
{ 1 2 3 4 5 }
2
LSPLT

results:
1: { { 1 2 } { 3 4 5 } }

Alternative Example (Method 2a)
{ 1 2 3 4 5 }
2
LSPLT

results:
2: { 1 2 }
1: { 3 4 5 }

Alternative Example (Method 2b)
{ 1 2 3 4 5 }
2
LSPLT

results:
2: { 3 4 5 }
1: { 1 2 }

Please respond with your preferred approach, which I will assume applies to both LSPLT and LRSPL.

I don't have strong feelings about any of the above methods; I see merits to all of them. As such, I will either keep or change those commands for the next release based on the feedback received. I do think it's important that both LSPLT and LRSPL use the same approach, though (whatever it ends up being).

Not to be disagreeable, but I have to vote for 2a for several reasons:

Firstly, if the list that LSPLT returns is broken out onto the stack using OBJ\-> DROP the result is the same as 2a.

Second, if the two lists in version 2a are combined using + the result will be the original list and if they are combined using 2. \->LIST the result will be the list-of-lists that LSPLT currently returns.

Finally, version 2a is what GoferLists' Split command returns and I can see no need for the two commands to be opposite.

IMHO, version 2a would result in less stack manipulation than version 2b in most cases.

John

I agree with 2a this makes most sense to me

[Gilles59]

(09-29-2017 04:45 PM)DavidM Wrote:  I was going to post this as a poll, but quickly realized that the only way to create a poll is to start a ght be to return the result simply as two separate lists i
Alternative Example (Method 2a)
{ 1 2 3 4 5 }
2
LSPLT

results:
2: { 1 2 }
1: { 3 4 5}

I vote 2a

I dont like 2b, looks counter intuitive for me as john explain. 2a works like OBJ-> etc.