HP 50g using datasets from SD card

[Cristi Neagu]

Hello,

If i have some amount of data, too large to be entered by hand, how could i go about creating a dataset on the SD card, and then reading it from there? What format should i be using for that? I tried looking this up but i'm not even sure where to start.

I tried generating a matrix on the calculator, exporting it to SD card, in an attempt to reverse engineer it. Normal text editors and IDEs only see it as a binary file. I then tried Debug4X, but either i can't figure out how to use it or it imports it as a binary file too. So does HPUserEdit.

So any tips? What program can i use to edit and create datasets that the calculator can read?

Thank you.

[cdmackay]

This thread suggests that you can RCL the contents of a file from SD to the stack.

But you can't go the other way, as you've found since things gets written to the SD card as binary-transferred objects.

Not sure if that thread helps…

[DavidM]

You may find it possible to put your data in an ascii file on the SD card, and then recalling the data as a string onto the stack. You could then execute the RPL command STR→ to convert it into something that could then be accessed in a program, either iteratively or by index.

If your data is a series of numbers, you could perhaps use a list format or possibly an array or matrix format as appropriate. You may need to split the data up into several different string objects, depending on how much memory is required for the combination of the ascii representation + the converted data.

Some general formats for possible data formats are as follows.

LIST: "{ obj1 obj2 obj3 ... }"
ARRAY: "[ num1 num2 num3 ... ]"
MATRIX: "[ [ num1 num2 num3 ] [ num4 num5 num6 ] [ num7 num8 num9 ] ]"

Once compiled by STR→, the object could be stored in native format as needed to a variable (either in main memory, flash, or SD card).

While it's possible to write the data in the appropriate binary format (the object formats are fully documented), it would almost certainly be easier to just have the calculator do it for you with STR→.

In any event, you will be limited by the amount of addressable memory in the calculator, which may end up being your largest obstacle if the dataset is very large. How large is the dataset you are converting?

[Cristi Neagu]

(05-17-2020 01:19 AM)cdmackay Wrote:  Not sure if that thread helps…

It does. Thank you.

(05-17-2020 01:21 AM)DavidM Wrote:  While it's possible to write the data in the appropriate binary format (the object formats are fully documented), it would almost certainly be easier to just have the calculator do it for you with STR→.

Do you happen to know where i can find this documentation?

(05-17-2020 01:21 AM)DavidM Wrote:  In any event, you will be limited by the amount of addressable memory in the calculator, which may end up being your largest obstacle if the dataset is very large. How large is the dataset you are converting?

The datasets would probably end up being on the order of 10-20kB. I did imagine that size would be an issue, but i wanted to get a proof of concept going for starters.

In any case, your suggestions led me to an ASCII to binary converter. I'll experiment with that and see if i can get it working. Thank you.

[DavidM]

(05-17-2020 01:40 AM)Cristi Neagu Wrote:  Do you happen to know where i can find this documentation?

Here's one source that includes detailed descriptions of RPL object formats: Introduction to Saturn Assembly Language.

Is this dataset a series of numerical objects? If so, a list or array might be good for potential targets. A list is nice if the data simply needs to be read from start to finish, but an array might be better if the data needs to be accessed by index.

This sounds like a fun project -- let us know how it goes!

[Cristi Neagu]

(05-17-2020 02:30 AM)DavidM Wrote:  

(05-17-2020 01:40 AM)Cristi Neagu Wrote:  Do you happen to know where i can find this documentation?

Here's one source that includes detailed descriptions of RPL object formats: Introduction to Saturn Assembly Language.

Is this dataset a series of numerical objects? If so, a list or array might be good for potential targets. A list is nice if the data simply needs to be read from start to finish, but an array might be better if the data needs to be accessed by index.

This sounds like a fun project -- let us know how it goes!

Thank you for the link.

This started from wanting to find ways i can use the 50g at work, even if they might not be practical solutions. And as a first goal, i want to get a bunch of data read and see if i can generate a histogram with it. I'm just starting out, so updates would be few and far between.

[John Keith]

Expanding on what David said, If your data is in the form of text files (.txt or .csv) it is probably easier to move the data as a flat list. The flat list can be easily converted into an array if that is desired.

For example if you have a file of 2500 numbers that represent a 50 x 50 matrix, add "{" to the beginning of the file and "}" to the end. Then OBJ-> or STR-> will return a list. Then, AXL { 50 50 } RDM will convert the list into a 50 x 50 matrix.

Additionally, a lot of this is easier to do on EMU48 if you have a windows PC. You can transfer strings or text files to the emulator with EDIT..COPY STACK and EDIT..PASTE STACK. You can then transfer files to the physical calculator with CONN4X or the SD card.

[Cristi Neagu]

(05-17-2020 09:37 PM)John Keith Wrote:  Expanding on what David said, If your data is in the form of text files (.txt or .csv) it is probably easier to move the data as a flat list. The flat list can be easily converted into an array if that is desired.

For example if you have a file of 2500 numbers that represent a 50 x 50 matrix, add "{" to the beginning of the file and "}" to the end. Then OBJ-> or STR-> will return a list. Then, AXL { 50 50 } RDM will convert the list into a 50 x 50 matrix.

That is a very straightforward method. I think even I can turn that into a program

[brickviking]

To add to this discussion, let's assume I have a SD card in my computer that I'm able to write to. What would be a C (or other language) program for writing that aforementioned array of 50x50 objects to a file on the SD card, so it can then be read back natively by the 50G reading the file in from the SD card? Or am I chasing up the wrong corridor here?

(Post 333)
(not evil. Honest. Truly! Seriously not evil!)

[cyrille de brébisson]

Hello,

The easiest way would probably to be to save the data as a HP native string.
To do so, you will need to prepend 5 bytes before your data which represents the string "structure". 5 nibbles for the string prologue (I do not remember it from the top of my head) and 5 nibble size (in nibbles = nbChrs*2+5)...

Harder would be to save it as a native matrix. BUT the loading would be quicker as you would not need to do a str-> on the calc after loading.

Cyrille

[grsbanks]

IIRC, if you just save it as a text file, the HP 50g interprets that as a string anyway.

[John Keith]

(05-19-2020 06:08 AM)grsbanks Wrote:  IIRC, if you just save it as a text file, the HP 50g interprets that as a string anyway.

Yes, it does. Cyrille does make a good point however, reading a large string with OBJ\-> or STR\-> is not very fast. Nor is the internal form of a matrix easy to deal with.

Since these files are already on a computer, it may be easiest to read them in with EMU48 and then save them as HP 50g matrices on the SD card. Then the files will be available on the 50g without conversion being necessary.

[DavidM]

(05-19-2020 05:16 PM)John Keith Wrote:  Since these files are already on a computer, it may be easiest to read them in with EMU48 and then save them as HP 50g matrices on the SD card. Then the files will be available on the 50g without conversion being necessary.

FWIW:

I tried converting a list of 1500 random real numbers that were generated on a PC using Excel. The list was made by copying a column of numbers into a text file on the PC, which I named "NUMBERS.TXT".

NUMBERS.TXT was copied to my 50g's SD card.

The SD card was inserted into my 50g, then the calculator was turned on.

I recalled the file/string to the stack using the Filer, and then enclosed the entire string in {} brackets with:

Code:

"{" SWAP + "}"  +

The resulting string was converted to a list with STR→ (which took about 28 seconds), then an array (with AXL). I then converted the array into a single-column array with:

Code:

{ 1500 1 } RDM

The resulting 1-column array was stored on the SD card as a binary object with:

Code:

"ARRAYOBJ" 3 →TAG STO

The binary object on the card had a size of 12,023 bytes. Similar steps using Emu48 had the same final result, but completed much faster (of course).

This text-based narrative looks much more involved than it actually was when I was doing it.