Calculators that allow direct operations on data stored on persistent storage?
|
02-18-2018, 03:00 PM
(This post was last modified: 02-18-2018 03:01 PM by pier4r.)
Post: #1
|
|||
|
|||
Calculators that allow direct operations on data stored on persistent storage?
Maybe the title is not so clear, I give some more info. After having written the wall of text below, I am not sure I clarified the problem :/ .
We know that physical calculators (or mobile systems in general) have little ram if one wants that the battery lasts long. Normally the ram is some kilobytes in size . 64kb for the casio 9860 and primz, 270kb for the 50g with userRPL, ~350kb + with the 50g with newRPL, some megabytes for ti nspire and hp prime, etc... If one is doing some little data collection over the days, say 500bytes per day, the ram won't be enough after a while. Especially if one wants to pack the data collected in one object, say a list. In this case I am ruling out the option that one is doing a workaround with partial lists that then have to be packed together somehow. In other words I am ruling out the option that the user himself is going to create an ad hoc solution to move "pages" of data between the ram and the persistent storage. Therefore would be neat to have the possibility to modify directly the data while this is residing on the persistent memory (internal storage or external storage like SD cards), without the need to recall all the data in memory. That is, the operating system of the calculator (or a library) is going to work for the user to recall only part of the data, modify it, and put it back. So far I know that only the newRPL allows this. Do you know any other OS or OS+libraries that allow to modify data stored on the persistent storage without recalling the entire data object in ram? That is, allowing to create data objects that may be larger than the available ram (but obviously smaller than the available storage). To make an example I mean the following. I have a data object of 20kb. I add 1kb of data more, now it is 21kb. To do the addition, instead of recalling the object in ram (occupying 20kb of ram) I recall only a fraction of it (using less ram than 20kb). At the end of the operation the data object on the persistent storage occupies 21kb containing the added content. On the 50g with userRPL , whether I use port1 (ram), port2(flash) or port3 (sd card) I always need to recall first the entire object in ram, so I need to have enough ram free, work on it, and then save it again. I have an object that is growing almost every day on the 50g. For the moment it is 21kb, and I need to have twice-three times its size of ram (port0) free for additional operations because I cannot edit directly the object saved on the persistent storage (at least not if I use the hp 2.15 firmware). For my predictions, unless I develop a paging solution, at the end of this year I may run out of ram. Hence the question. Sure one can say "use a prime that has megabytes of ram", but I'd like to know in general what is the status of the produced calculators and their software for data collection. (and at the end I will likely port the data on the newRPL OS) Wikis are great, Contribute :) |
|||
02-18-2018, 04:12 PM
Post: #2
|
|||
|
|||
RE: Calculators that allow direct operations on data stored on persistent storage?
If you mean calculators with internal or plug-in memory that can be treated like a file system, then the HP-41 with expanded memory (XM) or a HEPAX memory module can do the trick. Diego's NoV-64 provides four 4K pages of HEPAX RAM that can be bank switched with two other sets of four pages, all in persistent ferroelectric RAM. The 41CL has 128 pages of static RAM that can be plugged in and formatted as HEPAX RAM.
~Mark Remember kids, "In a democracy, you get the government you deserve." |
|||
02-18-2018, 07:32 PM
Post: #3
|
|||
|
|||
RE: Calculators that allow direct operations on data stored on persistent storage?
(02-18-2018 03:00 PM)pier4r Wrote: I have an object that is growing almost every day on the 50g. For the moment it is 21kb, and I need to have twice-three times its size of ram (port0) free for additional operations because I cannot edit directly the object saved on the persistent storage (at least not if I use the hp 2.15 firmware). For my predictions, unless I develop a paging solution, at the end of this year I may run out of ram. I would think that most all calculators will have some inherent limitations to the size of discrete objects that can be stored and manipulated. While it's possible that custom code could be created that might provide the ability to append new data to an existing object in storage, I'm wondering if that's really the best approach. Perhaps you could simply use a different format for the data you're storing which would require less space. Consider the following... Entering a list of the numbers 1-10 from the command line of the 50g creates an object as follows: (Exact mode) { 1 2 3 4 5 6 7 8 9 10 } Size: 34 bytes (Approx. mode) { 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. } Size: 38 bytes Coding the above as a string of successive 2-digit numbers (no spaces) results in this (mode doesn't matter): "01020304050607080910" Size: 25 bytes The string is smaller, but the first two are relatively close because those numbers are actually available as ROM objects, which means that each entry only takes 2.5 bytes/element. But watch what happens if we change to a different set of 2-digit numbers: (Approx. mode) { 50. 51. 52. 53. 54. 55. 56. 57. 58. 59. } Size: 110 bytes (Exact mode) { 50 51 52 53 54 55 56 57 58 59 } Size: 70 bytes "50515253545556575859" Size: 25 bytes Clearly we begin to see that there could be a substantial savings in storage space required if we store the data in string format. Yes, there would now be overhead to translate the data to/from the required string format, but as long as the discrete data elements are all the same size, we can access each one individually as needed by computing its offset. Further refinements could be made, of course, depending on the domain/range of the data you need to store. Storing the data in hex strings would provide even greater density of data if the individual numbers fit in the range 0-255. Is there a possibility of using a different format for the data you are storing that would require less space? |
|||
02-18-2018, 08:40 PM
Post: #4
|
|||
|
|||
RE: Calculators that allow direct operations on data stored on persistent storage?
David thanks for the info! Interesting the string approach.
(02-18-2018 07:32 PM)DavidM Wrote: Is there a possibility of using a different format for the data you are storing that would require less space? I strongly believe that my space usage on the 50g could be optimized, nonetheless I also wanted to use my "use case" to generalize the requirement. Therefore I asked whether there is a calculator that can offer to the user an almost "transparent" editing of files. Even when those are in the storage; without recalling them in memory, at least not entirely. For my specific case (that anyway I hope does not hinder the generic question above), the data format should satisfy several requirements given my needs and constraints. While I may use string, that from your examples are neat, I will need to develop some tooling (that is: helper programs) to convert and use the data if I want to do quick investigations, as packing a string without spaces is normally less readable than a RPL list (at least by my readability standards). Unfortunately I do not have much time to develop helper routines, otherwise I could try, along with strings, also "paged lists" or some incremental saving of sort. ( they may create an interesting project for later) For the moment, given several reasons, a list is the solution that fits better my need. If I ever find time, after having done some tasks that are waiting since years, I guess I may do incremental lists keeping most of the data on the SD. (or I may consider string as well) The original question is nonetheless interesting, because an OS that sees the storage as an "active" filesystem (and not only for archives) would make the life easier to the user that may focus on the data collection and not on the creation of helper routines (that although fun and challenging, require time). Wikis are great, Contribute :) |
|||
02-19-2018, 03:31 AM
Post: #5
|
|||
|
|||
RE: Calculators that allow direct operations on data stored on persistent storage?
The HP-71 does what I think you're looking for. Files reside in RAM (not flash or EEPROM), and can be expanded or contracted easily, and it's pretty transparent from the user's perspective. I have an extra 160KB of RAM in mine, 96KB in a card-reader-port module, and 64KB in a front module. (I would like to have gotten the full 160K in the card-reader-port module, the maximum available at the time, but it was outside my budget.) Each module goes in 32KB "ports," however many it takes to fill up the capacity of the module. Any memory port that is "claimported," ie, made part of the basic RAM, can be used directly for variables too, not just files. Claimported memory is not divided into 32K sections. They're just all run together.
Since I use my 71 so little now (as I mostly just use my 41cx), the 71's batteries last many years with all this additional RAM in it. CMOS RAM's power-supply current when it's in standby mode is only the leakage current, meaning probably picoamps or femptoamps. In my work in about 1986, I noticed an 8KB SRAM IC holding its data from one day to the next with only a 10uF capacitor for a power supply. If the current were even 1uA, a 10uF capacitor would go dead in ten seconds. I did not stretch it out to see how long it would go though, like a week, month, or whatever. http://WilsonMinesCo.com (Lots of HP-41 links at the bottom of the links page, at http://wilsonminesco.com/links.html#hp41 ) |
|||
02-23-2018, 10:09 PM
Post: #6
|
|||
|
|||
RE: Calculators that allow direct operations on data stored on persistent storage?
David made a good point: if you need compact storage, you need to find the optimum format. However, if you need fast access, you have to work with the data in whatever format the CPU or the program uses, you can't afford to convert back and forth all the time.
There's always a compromise in between, and you'd be surprised how much data you can fit. I vaguely remember the data collector from Precision Solutions, LLC, it could store 7000 points (as in surveying coordinate points) in unused RAM areas of the 50g, you were able use the calculator normally (well, as a data collector) and all those points were "hidden" from view. It's all a matter of finding the proper format. I think you want some kind of virtual memory management. With VMM you CAN have the whole thing in RAM, or at least you think you do, while the operating system swaps data to disk in/out as you use your data. The storage format is still very relevant, though. For example, in a list of variable size objects (which is your preferred method of storage), you'd have to skip N objects to access item N, which implies reading the list from start to N for every access (swapping it in and out of memory). On the other hand, if you had fixed records you could compute the offset to object N and only read that one record from storage. Helper routines would be the wiser way to go, I think, otherwise you'll pay the price in performance if you manage large lists. You should probably start thinking of your "paging" solution before you run out of space. One possible solution is for example to store 100 records per list, and store the lists in variables/files 'Lmmm', Where mmm is the higher digits of the record index: Index mmmNN is stored in list variable 'Lmmm', list index NN. It is simple that only takes you a few minutes to write the helpers, relatively fast, and you can move 'Lmmm' files to SD card when they become "cold" and are no longer modified. |
|||
02-24-2018, 11:15 AM
(This post was last modified: 02-24-2018 11:22 AM by pier4r.)
Post: #7
|
|||
|
|||
RE: Calculators that allow direct operations on data stored on persistent storage?
(02-19-2018 03:31 AM)Garth Wilson Wrote: The HP-71 does what I think you're looking for. Files reside in RAM (not flash or EEPROM), and can be expanded or contracted easily, and it's pretty transparent from the user's perspective. Yes exactly, thanks for the info. Anyway some limitations on my side: no 71B (and no budget for it), the ram in the 50g is already of the same size, and no "modern" connection with a computer as far as I know. Anyway the concept is exactly that, more or less. (02-23-2018 10:09 PM)Claudio L. Wrote: I think you want some kind of virtual memory management. With VMM you CAN have the whole thing in RAM, or at least you think you do, while the operating system swaps data to disk in/out as you use your data.Yes, thanks for finding the technical term. Also thanks for your post in general. Quote:You should probably start thinking of your "paging" solution before you run out of space. Yes for the info that I got so far, yes. In my case performance is not that of an hassle . As long as I can store the data in the object within 30 seconds. I am willing to wait 30seconds, other people maybe not. If I understood correctly newRPL has a sort of virtual memory management, or anyway I can work on the data that is directly stored on the SD card without being completely recalled in ram, right? And sure also the format is important, as in every case the solution has to fit the available resources (deadline, time, skill, willpower, hardware, etc.). I guess I will slowly refactor from using a directory of list on the RAM, to a directory of incremental lists on the SD card, with the last increment in ram. Similar to what Claudio suggested. String & co would be maybe better (see space optimization) but then the helper routines may grow larger and I am not sure whether I will have the time to maintain them every time I find a bug. For example my data collection started in June 2017 (I mentioned it sometimes in other topics) and the directory was like 2-3kb. I predicted 80-100 bytes more every day, but my prediction is not really holding, or better it is more complicated. For the moment I collected in every category about 250 points. The directory (included backup routines, that changed over time as I fixed them) is 22 kilobytes. The average is 88 bytes added per day but actually the data added recently is bigger than before because I added some new categories. The helper routines for the moment require at least 2 times the space free to be sure that no conflicts arise (because the code is not extra refined). So I need in total 66kb. So trying to overestimate (but not too much), with 250 bytes added per day, I will have 113 kb at the end of the year and then game over, since I have around 200 kb of ram free for this task. Therefore I asked about calculators with this sort of "transparent" storage. If I use an optimized format like strings, I may delay the problem but then it will happen eventually. I guess I need to start to do the paging solution, to keep the data in ram under 30kb. I also thought to just save the data on the pc every now and then and reset the counters on the 50g, but in theory I'd like to analyze the data on the 50g itself too, so while I backup it on the pc, I don't want to move it from the calculator. Another thing I discovered is that the backup on the SD card, in form of HPDIR, is not easy to read on the pc. So I need to convert all the full backups of the directory (one for every day, so I have 250 of them) in string format to be able to read them easily on the pc. the data is like the following (yes somewhere I stored integers instead of reals, I was in the wrong mode). I'll convert them eventually (code makes the panel too long on the horizontal axis :/ ) Quote:{ 8.062017 9.062017 10.062017 11.062017 12.062017 13.062017 14.062017 15.062017 16.062017 17.062017 18.062017 19.062017 20.062017 21.062017 22.062017 23.062017 24.062017 25.062017 26.062017 27.062017 28.062017 29.062017 30.062017 1.072017 2.072017 3.072017 4.072017 5.072017 6.072017 7.072017 8.072017 9.072017 10.07207 11.072017 12.07207 13.072017 14.072017 15.072017 16.072017 17.072017 18.072017 19.072017 20.072017 21.072017 22.072017 23.072017 24.072017 25.072017 26.072017 27.072017 28.072017 29.072017 30.072017 31.072017 1.082017 2.082017 3.082017 4.082017 5.082017 6.082017 7.082017 8.082017 9.082017 10.082017 11.082017 12.072018 13.082017 14.082017 15.082017 16.082017 17.082017 18.082017 19.082017 20.082017 21.082017 22.082017 23.082017 24.082017 25.082017 26.082017 27.082017 28.082017 29.082017 30.082017 31.082017 1.092017 2.092017 3.092017 4.092017 5.092017 6.092017 7.092017 8.092017 9.082017 10.092017 11.092017 12.092017 13.092017 14.092017 15.092017 16.092017 17.092017 18.092017 19.092017 20.092017 21.092017 22.092017 23.092017 24.092017 25.092017 26.092017 27.092017 28.092017 29.072017 30.092017 1.102017 2.102017 3.102017 4.102017 5.102017 6.102017 7.102017 8.102017 9.102017 10.102017 11.102017 12.102017 13.10217 14.102017 15.102017 16.102017 17.102017 18.102017 19.102017 20.102017 21.102017 22.102017 23.102017 24.102017 25.102017 26.102017 27.102017 28.102017 29.102017 30.102017 31.102017 1.112017 2.112017 3.112017 4.112017 5.112017 6.112017 7.112017 8.112017 9.112017 10.112017 11.112017 12.112017 13.112017 14.112017 15.112017 16.112017 17.112017 18.112017 19.112017 20.112017 21.112017 22.112017 23.112017 24.112017 25.112017 26.112017 27.112017 28.112017 29.112017 30.112017 1.122017 2.122017 3.122017 4.122017 5.122017 6.122017 7.122017 8.122017 9.122017 10.122017 11.122017 12.122017 13.122017 14.122017 15.122017 16.122017 17.122017 18.122017 19.122017 20.122017 21.122017 22.122017 23.122017 24.122017 25.122017 26.122017 27.122017 28.122017 29.122017 30.122017 31.122017 1.012018 2.012018 3.012018 4.012018 5.012018 6.012018 7.012018 8.012018 9.012018 10.012018 11.012018 12.012018 13.012018 14.012018 15.012018 16.012018 17.012018 18.012018 19.012018 20.012018 21.012018 22.012018 23.012018 24.012018 25.012018 26.012018 27.012018 28.012018 29.012018 30.012018 31.012018 1.022018 2.022018 3.022018 4.022018 5.022018 6.022018 7.022018 8.022018 9.022018 10.022018 11.022018 12.022018 13.022018 14.022018 15.022018 16.022018 17.022018 18.022018 19.022018 } Wikis are great, Contribute :) |
|||
02-24-2018, 03:54 PM
Post: #8
|
|||
|
|||
RE: Calculators that allow direct operations on data stored on persistent storage?
I have no experience with the SD card on a 50g yet so I can't help you there. However, there is a function on the HP 50g that does what you are looking for.
Key in 97.01 [ENTER] then key in MENU [ENTER]. You will be warped to the data menu in the STATS soft menu. Next type in another entry to your record in a matrix format via matrix writer or with the matrix brackets. Next press the ∑+ soft label on the F1 key. This will add another entry to the data in ∑DAT without recalling the contents. Test it out with a little bit of data. Unfortunately for you, your data seems to be in lists. So transferring to this method is going to be a pain. Also, the data has to be in a variable called ∑DAT which is inconvenient. Plus, I don't know if this can work with data stored on an SD card. Still, this method is work mentioning because maybe it can work with an SD card. Do you understand what I'm explaining here? |
|||
02-24-2018, 07:50 PM
(This post was last modified: 02-24-2018 08:01 PM by DavidM.)
Post: #9
|
|||
|
|||
RE: Calculators that allow direct operations on data stored on persistent storage?
There's a case to be made for simply breaking down your data into smaller "chunks". Those chunks could be based on some logical grouping, such as months or weeks. I'm not entirely sure what your data represents, but it appears that it might be something along the lines of 19 data points for each designated day. If that's the case, 31 day's worth of data would consume about 5K in uncompressed format.
While not a space saver, you might also consider using arrays to store the data chunks instead of lists. Some of the built-in commands for arrays might be very useful for you. Converting the data from your existing format wouldn't be too difficult using AXL along with some of the ListExt commands. Just to give you an idea of one possible way to do that, I copied your lists onto the stack of an emulated 50g and then performed the following steps: Code: 20 →LIST The result is a 257-row by 20-column array with the dates in the first column and each subsequent column containing the apparent data values for those dates. While there's no space savings in doing this, it does make it relatively easy to extract the data of a specific day if you know its offset in the array: Code: 13 ROW- NIP ...yields the following in stack level 1: Code: [ 20.062017 5. 4. 0. 2. 4. 3. 3. 4. 2. 3. 0. 4. 4. 3. 4. 2. 0. 0. 360. ] Extracting columns for date ranges and storing as ΣDAT would allow you to use some of the statistical functions that Carsen was mentioning. If you prefer the data in list format, AXL will convert the given row (or column) into a list. It would then be a simple matter, for example, to compute the average of column 20 over a given range of rows. Finally, I know you're not interested in trying to use an alternate format for the data right now, but if you should ever reconsider that, here's a couple of thoughts: 1) It looks as though the first list may be a series of dates. Furthermore, it appears that those dates may actually be in sequence. You may not even need to notate the dates at all, and simply rely on the relative positions of the data elements to imply the date instead. For example, the first data element in each list may be thought of as being associated with the "base date" of 2017-06-08, the second being associated with 2017-06-09, etc. Determining the date for a given column is then a simple computation of "8.062017 (row-1) DATE+". If all dates aren't actually in the list (I didn't inspect each one for gaps), you could still save some space by saving the offset from your base instead of the actual date itself. In other words, 0 instead of 8.062017, 1 instead of 9.062017, ... 256 instead of 19.022018, etc. 2) It looks like all but the first and last lists contain only single digit numbers. Couldn't those be combined easily into groups of (say) 10 at a time? Using the above row as an example, the first entry would be "5402433423". That would cut down considerably on the storage requirements. And since I know you have the ListExt library, it would be simple to extract the individual values from the combined number. Just providing more things to think about! |
|||
02-24-2018, 08:21 PM
Post: #10
|
|||
|
|||
RE: Calculators that allow direct operations on data stored on persistent storage?
(02-24-2018 11:15 AM)pier4r Wrote:(02-19-2018 03:31 AM)Garth Wilson Wrote: The HP-71 does what I think you're looking for. Files reside in RAM (not flash or EEPROM), and can be expanded or contracted easily, and it's pretty transparent from the user's perspective. There is the PIL Box for the HP-IL connection to a PC, but I know almost nothing about it. (You can search for it on this forum.) You can go a long ways in interfacing if you go a long ways in spending money too though. I have the HP-IL interface converters to RS-232, parallel, IEEE-488, and 80-column monochrome video. In the mid-to-late 1980's at work, I used 71B-Talk, software for both the 71 and a PC to link them and use the PC's keyboard as a full-sized keyboard for the 71, use the PC's monitor as a monitor for the 71, and I believe you could transfer files and so on too. I don't remember any details. I believe it used an RS-232 interface. http://WilsonMinesCo.com (Lots of HP-41 links at the bottom of the links page, at http://wilsonminesco.com/links.html#hp41 ) |
|||
02-24-2018, 09:04 PM
Post: #11
|
|||
|
|||
RE: Calculators that allow direct operations on data stored on persistent storage?
(02-24-2018 11:15 AM)pier4r Wrote: If I understood correctly newRPL has a sort of virtual memory management, or anyway I can work on the data that is directly stored on the SD card without being completely recalled in ram, right? It doesn't have virtual memory in the sense you want. It does use the memory management unit (MMU) to assign pages of ram to grow stack and storage regions independently. This makes better use of the available RAM, but does not swap anything to SD card, so when you are out of memory, you are also out of luck. |
|||
03-01-2018, 06:15 PM
Post: #12
|
|||
|
|||
RE: Calculators that allow direct operations on data stored on persistent storage?
Here's an example of some helper routines that would translate some of your data back and forth between a list format and storage format. The ListExt library is required for several of the commands. I realize that your actual data doesn't match the format I'm using here, but I wanted to show several different examples in these two routines to illustrate how the conversions could be done.
In the above example, a vector was shown that appeared to have the data for a given day. I've converted it to a list for this example: Code: { 20.062017 5. 4. 0. 2. 4. 3. 3. 4. 2. 3. 0. 4. 4. 3. 4. 2. 0. 0. 360. } Using that same data as the starting point, the following code will convert the list to a string representation: Code: \<< The padding is required to make sure that the first and last entries always use a consistent number of digits. The result of the above routine with the specified data becomes "@[PXTS*M'H4iT7". That's not very useful in terms of readability, but for storage it's only 19 bytes compared to the 71 bytes used by the list. You could easily save even more by stringing together multiple day's worth of entries. The following is the reciprocal to the above: Code: \<< |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 2 Guest(s)