Search
Member List
Calendar
Help
HP Forums
/
HP Calculators (and very old HP Computers)
/
General Forum 
/
50g: copy all stored objects to SD card individually
/
50g: copy all stored objects to SD card individually
|
50g: copy all stored objects to SD card individually
|
|
10-17-2016, 08:32 AM
(This post was last modified: 10-17-2016 10:55 AM by matthiaspaul.)
Post: #21
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-16-2016 07:51 PM)DavidM Wrote: ...and yes, copying the text and pasting into a code block creates some confusion due to mixing code pages again.Thanks a lot! Unfortunately, I have difficulties explaining some of the differences between these two views - some seem to make sense in the first, some in the second view. Are you familiar with using a disk editor? I guess we could more reliably see what's actually going on under the hood by bypassing Windows and having a look at a hex dump of the on-disk structures. If you'd want to give it a try, I can recommend HxD, a small free disk editor for Windows: https://www.mh-nexus.de/en/hxd/ . In "Extras" you'd have to open a view of the FAT partition (drive I: in your case) as a logical volume, and then use "Search" to find f.e. the directory entry holding the string "MIXCASE". A hex dump of perhaps about hundred bytes before and five-hundred bytes after that location would be helpful. Greetings, Matthias -- "Programs are poems for computers." |
|||
|
|
10-17-2016, 04:42 PM
Post: #22
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-16-2016 05:30 PM)DavidM Wrote:(10-16-2016 05:24 PM)DavidM Wrote: Just tried this, ... So it seems it still creates some lowercase SFN, unless the SFN is stored as uppercase but displayed lowercase because the lowercase flag is set? What would directly accessing the short name do? DIR lcase_~1 DEL lcase_~1 |
|||
|
|
|
10-17-2016, 04:47 PM
Post: #23
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-16-2016 07:26 PM)matthiaspaul Wrote: :3:"SPECIÄL" STO no reason to create a LFN entry but some system may do There's one reason: The character >127. Creating an LFN is the only way to store it as Unicode. I modified newRPL to always create LFN's in cases like this, just to eliminate the ambiguity of using CP translation. Technically not needed if you are willing to put up with old CP mess, though. |
|||
|
|
|
10-17-2016, 05:04 PM
Post: #24
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-17-2016 08:32 AM)matthiaspaul Wrote: Are you familiar with using a disk editor? I guess we could more reliably see what's actually going on under the hood by bypassing Windows and having a look at a hex dump of the on-disk structures. Deep back in my history I created a disk imaging app for a large well-known computer company in the late '80s. So, yes, I've got some familiarity with disk editors (though not FAT-based, and it's been a while). I'll check out the app you mentioned and start digging around with it on the SD cards. I was already thinking of doing that anyway, as I've been trying to remember the sequence of things I did prior to the corruption of the SD card file system. I'd love to be able to recreate that so as to know what to avoid in the future. |
|||
|
|
|
10-17-2016, 05:21 PM
Post: #25
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-17-2016 04:42 PM)Claudio L. Wrote: So it seems it still creates some lowercase SFN, unless the SFN is stored as uppercase but displayed lowercase because the lowercase flag is set? Neither "DIR lcase_~1" nor "DIR LCASE_~1" succeeded, both failing with "File Not Found" (same with DEL, msg: "Could Not Find I:\<fn>"). Both "DIR lcase" and "DIR LCASE" succeeded for the other file. I'm starting to wonder if DIR /X is populating the LFN and SFN fields with presumptions in cases where the actual data is questionable. All the more reason to use the disk editor to look at the actual stored strings. And I'm still wondering about this same scenario on other platforms -- I don't have ready access to OS/X or *ux systems to see how this affects them. |
|||
|
|
|
10-17-2016, 06:15 PM
(This post was last modified: 10-19-2016 07:10 PM by matthiaspaul.)
Post: #26
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-17-2016 04:47 PM)Claudio L. Wrote:I agree, at least in this context. (I personally try to avoid *VFAT* LFNs whenever possible, for I consider the way they were retrofitted into the FAT filesystem a design "nightmare". There would have been more robust methods to integrate long file names into FAT while still maintaining full backward compatibility - and without speed or size penalties. But keeping FAT volumes free of VFAT LFN entries only makes sense for as long as the whole medium can be kept free of them realistically (for example on local partitions using partition IDs not recognized by systems using VFAT LFNs). In the case of an exchange medium of a calculator using a custom character set it would be unrealistic to maintain this goal. Actually, given that the RPL character set is based on ISO/IEC 8859-1 and thus many codepoints are identical to Unicode codepoints, being able to store characters in Unicode is highly desirable - assuming a bug-free implementation... ;-)(10-16-2016 07:26 PM)matthiaspaul Wrote: :3:"SPECIÄL" STO no reason to create a LFN entry but some system may doThere's one reason: The character >127. Creating an LFN is the only way to store it as Unicode. I modified newRPL to always create LFN's in cases like this, just to eliminate the ambiguity of using CP translation. Technically not needed if you are willing to put up with old CP mess, though. So, my comment wasn't meant to imply that this would be a bad idea, just that it is technically not required from the viewpoint of the filesystem. And in fact, as we can see from the DIR /X output above, the calculator does not create a LFN entry in this case. Greetings, Matthias -- "Programs are poems for computers." |
|||
|
|
|
10-17-2016, 06:26 PM
Post: #27
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-17-2016 04:42 PM)Claudio L. Wrote: So it seems it still creates some lowercase SFN, unless the SFN is stored as uppercase but displayed lowercase because the lowercase flag is set?I found another description matching our observation in a bug-report by James M. Prange for firmware 2.09: http://bugs.hpcalc.org/show_bug.cgi?id=235 Quote:Invalid "short filename" aliases including lower-case letters are sometimes used with "long filename" directory entries on the MMC/SD card. This doesn't seem to cause any problem on the calculator itself (the filer displays the alias in all-capitals), and MS Windows Explorer correctly displays the long filename, butGreetings, Matthias -- "Programs are poems for computers." |
|||
|
|
|
10-17-2016, 07:29 PM
Post: #28
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-17-2016 08:32 AM)matthiaspaul Wrote: A hex dump of perhaps about hundred bytes before and five-hundred bytes after that location would be helpful. It appeared that all of the directory entries existed in a single sector for this particular scenario. Using a width of 32 bytes seemed to nicely align with entries, so I used that for the following output: Code: Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F |
|||
|
|
|
10-20-2016, 03:23 AM
(This post was last modified: 10-20-2016 04:46 AM by DavidM.)
Post: #29
|
|||
|
|||
|
RE: 50g: copy all stored objects to SD card individually
The same test case, using Claudio's SDSto routine from SDLIB to store the files:
Code: Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1FSo it appears that all-lower-case-or-high-ascii-names-that-fit-within-8.3 object names get UPCASEd by the version of SDLIB included with SDFiler, and with no LFN entry created. Mixed case gets the LFNs. I'm assuming that all longer-than-8.3 names end up with the LFNs as well. All of which leaves my head spinning with regards to a reasonable path forward for the treatment of object names for this app, which was on shaky ground from the start. Given that the number of people in the entire world that could make use of such an app is probably countable on one hand (possibly one finger), this seems like more effort than it's worth at this point. The amount of manual manipulation of files, object names, and cross-checking is not much different than simply copying the files with existing tools. The calculator doesn't track modification dates for individual objects, so relying on the SD card file system dates isn't particularly meaningful (those dates only tell you when the file was written, not when the actual object was updated). Encapsulating the real names as a string within a list object is a concept which could be useful, but IMHO doesn't add enough value to this app's concept to pursue. But if nothing else, perhaps the discussion has had some value for NewRPL's implementation.  Thanks to Matthias, Claudio and 3298 for their participation! |
|||
|
|
|
10-20-2016, 07:11 PM
Post: #30
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-20-2016 03:23 AM)DavidM Wrote: The same test case, using Claudio's SDSto routine from SDLIB to store the files: Don't forget that the name is stored as uppercase (as it should be) but there's a flag telling the OS if the actual name is uppercase, lowercase, or first-letter-uppercase-only. |
|||
|
|
|
10-20-2016, 08:56 PM
Post: #31
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-20-2016 07:11 PM)Claudio L. Wrote: Don't forget that the name is stored as uppercase (as it should be) but there's a flag telling the OS if the actual name is uppercase, lowercase, or first-letter-uppercase-only. I haven't, but it appears that condition doesn't apply when a high-ascii character is encountered in the object name. Either that, or Win7 Explorer and dir/x aren't respecting it for the directory entries at the bottom of the list. For what it's worth, the following SysRPL code generated those directory entries: Code: BINT0 BINT1 %0 "lcase" BINT1 SDLIB DROP I'm not being critical of that design choice -- there aren't any "easy" answers for the best approach here. But it seems clear that there is a good likelihood that object names will get mangled enough to require extra (manual) effort on the user's part to make sure they organize their files properly, which negates some of the marginal value an app like this would provide. Perhaps if I thought that I would be likely to use it, I'd see more value in it. It turns out that I almost never create programs on the calculator itself, so it's actually pretty rare for me to have a need to extract objects from the calculator... I almost always go in the opposite direction (computer -> calculator). When doing so, individual objects are usually sent via wire. Wholesale restore operations (main and port memory) are handled via custom apps loaded from the SD card. This makes it fairly easy for me to replicate the exact same configurations for my real calcs and the Emu48 counterparts.
|
|||
|
|
|
10-20-2016, 10:19 PM
Post: #32
|
|||
|
|||
|
RE: 50g: copy all stored objects to SD card individually
I have reformatted the SDLIB hex dump somewhat:
Quote:"lcase":All-lowercase name fitting into 8.3 scheme, so the case info is stored in the NT case bits. IIRC, bit 4 was meant for the extension, not the name, so shouldn't this read 08h instead of 10h? Quote:"lcase;;;;;;;":All but one semicolon stripped off by library code as they are not needed here. LFN entry created because ; is not a valid filename character in an SFN. Claudio, would the remaining ; have been stripped off as well, if the volume would not already hold another file named "lcase"? Quote:"Mixcase": Quote:"CamelC": Quote:"UPCASE":No LFN created as all-uppercase and still fitting into 8.3 scheme. Quote:"SPECIÄL":No LFN created as detected as all-uppercase and still fitting into 8.3 scheme. In general, if codepoint C4h will be considered as an uppercase letter depends on codepage and country settings, so other operating systems may treat it as "special character" and create a LFN entry instead. Quote:"speziäl":No LFN created as detected as all-lowercase and still fitting into 8.3 scheme. In general, if codepoint E4h will be considered as lowercase letter depends on codepage and country settings, so other operating systems may treat it as "special character" and create a LFN entry instead. This is a bit risky, because if E4h will be treated as lowercase letter on some system, it may not be able to access the file (as no lowercase letters are allowed in SFNs). Since this file has only a name and no extension, shouldn't the bit flag read 08h rather than 10h? Quote:"specÿal":Same here. Quote:"speci■l":Greetings, Matthias -- "Programs are poems for computers." |
|||
|
|
|
10-21-2016, 02:44 AM
Post: #33
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-20-2016 10:19 PM)matthiaspaul Wrote: I have reformatted the SDLIB hex dump somewhat: Yes, don't forget this is a very old binary, there were multiple bugs fixed after SDFiler was released. It's almost pointless to analyze this now. (10-20-2016 10:19 PM)matthiaspaul Wrote: Claudio, would the remaining ; have been stripped off as well, if the volume would not already hold another file named "lcase"?Yes, trailing semicolons are treated as a system feature. The system will remove them and/or ignore them on comparisons, and add them as needed on creation. The user can't create or see a file with a trailing ; It is possible for the user to create with another OS 2 files with identical name except for a trailing semicolon. hpgcc/newRPL will open the first one that matches in the current directory, both will match in searches so you'll see both in the directory, but as identical files. |
|||
|
|
|
10-22-2016, 06:53 PM
Post: #34
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-21-2016 02:44 AM)Claudio L. Wrote:Yes, but unfortunately this means that this version of SDLIB does not preserve the case properly for some types of filenames fitting into the 8.3 scheme, and so it won't be of much help for David...(10-20-2016 10:19 PM)matthiaspaul Wrote: IIRC, bit 4 was meant for the extension, not the name, so shouldn't this read 08h instead of 10h?Yes, don't forget this is a very old binary, there were multiple bugs fixed after SDFiler was released. It's almost pointless to analyze this now. Is there a newer version of SDLIB publically available somewhere? If so, for the benefit of the users the SDFiler package should be updated with the latest version of SDLIB, I think. (And once the SD/FS related stuff in newRPL is fully debugged, it might even make sense to backport this to SDLIB - not now, but perhaps in a year or so.) Regarding David's problem, both the original HP firmware and SDLIB seem to provide usable results for filenames not fitting into the 8.3 scheme. In the case of the original firmware we already know that it may still write buggy (lowercase) SFNs in this case, and that characters 80h..A0h are not translated into Unicode properly, but at least this happens in a predictable and therefore correctable way. So, enforcing the creation of LFNs by appending some long enough dummy string to shorter (or all) file names still appears to be a basis for a possible workaround, although not a particularly nice looking one... Greetings, Matthias -- "Programs are poems for computers." |
|||
|
|
|
10-22-2016, 07:30 PM
Post: #35
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-22-2016 06:53 PM)matthiaspaul Wrote: Yes, but unfortunately this means that this version of SDLIB does not preserve the case properly for some types of filenames fitting into the 8.3 scheme, and so it won't be of much help for David... ...but that's OK. I was already a bit dubious about putting any significant effort into this project, as with further investigation it seemed more like "a nail in search of a hammer" than a useful tool. Regardless of the reasons and tools used, it became obvious that copying individual files to the SD card was going to result in name conversions that would make the user have to spend extra time dealing with them. To me, that negated the benefit of the tool in the first place. (10-22-2016 06:53 PM)matthiaspaul Wrote: Is there a newer version of SDLIB publically available somewhere? If so, for the benefit of the users the SDFiler package should be updated with the latest version of SDLIB, I think. (And once the SD/FS related stuff in newRPL is fully debugged, it might even make sense to backport this to SDLIB - not now, but perhaps in a year or so.) I agree that it would be good to replace the version of SDLIB included with SDFiler once the charset conversions and file system handling are finalized. The modularity of the existing SDFiler package should make this fairly straightforward, provided the new SDLIB contains a similar interface. I have my doubts that Tim is very focused on such old products nowadays, but perhaps he wouldn't mind if someone else updated the library for him (he did provide the Debug4x source, after all). |
|||
|
|
|
10-22-2016, 07:57 PM
Post: #36
|
|||
|
|||
RE: 50g: copy all stored objects to SD card individually
(10-20-2016 03:23 AM)DavidM Wrote: Encapsulating the real names as a string within a list object is a concept which could be useful, but IMHO doesn't add enough value to this app's concept to pursue.Another idea along that same path, but different anyway: ;-) Are you familiar with "DESCRIPT.ION" files? These are (normally hidden) 8-bit text files holding one line of information for each file in the current directory - typically the information is just a descriptive string, but it could also be any other info like extended attributes or configuration data. These files are created and maintained alongside file operations by JPSoft's advanced command line processors (4DOS, 4OS2, 4NT, TC). Descriptions are created using the DESCRIBE command (or a normal text editor), and are shown similar to long filenames by DIR. Various third-party file managers and applications support them as well. Compared with storing the long filenames inside the files, storing the long names in DESCRIPT.ION files would have the advantage that the original files can remain unchanged and that they are easier retrievable. http://jpsoft.com/ascii/descfile.txt (basic format description) Greetings, Matthias -- "Programs are poems for computers." |
|||
|
|
|
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 7 Guest(s)