Is PACK on the HP-71 buggy?

[Joe Horn]

IN BRIEF: Does the PACK command in the HP-71 screw up LIF disk images mounted in ILPer when using a PIL-Box?

BACKGROUND: When a file is deleted on a LIF disk, it leaves a "hole" in the directory and data area (the file type is merely changed to 0000, but the file's name and contents remain on the disk). The PACK command on the HP-71 eliminates those holes by packing the directory and the data area, by shifting everything after the holes upwards, filling up the holes. After PACK, all available space is in a single block at the end of the directory, and in a single block at the end of the data area.

DISTURBING OBSERVATION: Using the PACK command with an HP-71 (rev 1BBBB; physical one, not emulator) and a PIL-Box (rev 1.5) and ILPer (ver 1.43) does more than just pack the disk image; it also writes bogus information into the directory area, almost without fail, even when no packing is needed.

EXAMPLE: I have a LIF disk image called "DIVISORX Lex.dat" whose hex contents originally began like this:



As you can see, the disk only contains two files (DIVISORX and DIVISORS), followed by the directory area, five entries of which (highlighted in yellow above) happened to contain zeroes instead of the usual FF's, but that shouldn't matter. The disk's directory size is 616, of which 614 are available. No packing of the directory is needed, since it contains no deleted files.

After mounting that disk with ILPer, and executing PACK on the HP-71 via a PIL-Box, that LIF image file's hex contents changed to this:



As you can see, five bogus directory entries magically appeared, AFTER the "end of directory" line at hex position 0240. These five files are nowhere on this disk; in fact (strange to tell!) they were on the previous disk which I PACKed! Apparently PACK leaves some directory data in RAM somewhere, and that old data gets written to the next disk that gets PACKed!

After doing this many times, to many different LIF images, I have observed that it doesn't matter whether or not there are any zero'd directory entries; bogus ones get written in both cases. The good news is that they always seem to get written after the end-of-directory marker... but the bad news is that if the directory is full (or almost full), the bogus entries MIGHT obliterate the beginning of the data area, which scares the heck out of me.

Notes: The PACKDIR command exhibits this same behavior. The number of bogus directory entries written is not always 5.

QUESTIONS: Is this a known and harmless artifact of PACK and PACKDIR, or is it a bug? If it's a known bug, is there a reliable workaround?

[Dave Frederickson]

Bug reference number 1184-1 has to do with packing files on a 9114, but that's a bug with the 9114, not the 71.

In packbug.txt, Michael Markov describes the behavior you're observing.

Dave

[Joe Horn]

Thanks, Dave! Since Michael Markov's writeup doesn't mention PACK destroying data outside the directory area, I'll stop worrying about that.

[rprosperi]

(07-14-2014 11:11 PM)Joe Horn Wrote:  QUESTIONS: Is this a known and harmless artifact of PACK and PACKDIR, or is it a bug? If it's a known bug, is there a reliable workaround?

I have read quite recently somewhere? (likely an old forum post, but possibly elsewhere), while researching using the 9114 with the 71B that the PACK and PACKDIR commands are known to have consistent problems with the 9114A, less consistent but still occaisional problems with the 9114B, and ultimately that the author believed the real problem was in the 71 and basically suggested to avoid using these commands; a better solution is to create new media copies if the original media is that discombobulated. The prior disk data that you observed cached in the 71 could be indicative of the problem alluded to here.
(Any chance that old directory data is cached in ILPer?)

That said, when I initially got my PIL-Box, I created lots of LIF files, copied files, deleted them, added, recopied, etc. to learn the devivce's behavior and also tried PACK and PACKDIR and saw no ill effects.

I would not call my testing exhaustive by any means, but after these PACK commands, all files were still intact, from a CATalog viewpoint; I did not actually load and test each file to be sure.

Probably the worst case scenario is packing a volume whose directory has only 1 or 2 open slots. The bogus entries might conceivably get written past the end of the directory table and into the file data area?

Curious - how did you discover this?

[Tim Wessman]

(07-15-2014 06:13 PM)rprosperi Wrote:  Curious - how did you discover this?

He is Joe Horn. Is any other explanation needed?

[Dave Frederickson]

(07-15-2014 09:01 PM)Tim Wessman Wrote:  

(07-15-2014 06:13 PM)rprosperi Wrote:  Curious - how did you discover this?

He is Joe Horn. Is any other explanation needed?

You speak of him as if he were a guru. Others of that status have unlikely-superhero names such as:

(05-20-2014 01:51 AM)Sylvain Cote Wrote:  

(05-13-2014 05:19 PM)John W Kercheval Wrote:  Name your top 5 HPCalc Gurus & why....

Code:

Angel Martin            Master of the NUT, creator of so many ROM's and the one who always pushes the boundaries of the 41 farther away
Diego Díaz              Creator of the Clonix & NoV modules that allow us to use the marvels from Angel. 
Monte Dalrymple         Creator of the ultimate HP-41, the 41CL who moved the 41 into another dimension.
Jean-Francois Garnier   Master of the Loop and the one who connected the old generation (IL) to the new one (USB) with it's PIL-Box.
Christoph Gießelink     Mr. Simulator, who took the IL simulator work of JFG for MS-DOS, moved it to Windows, refined it and expanded it beyond dreams.

Might I suggest "Professor Titan".

Dave

[rprosperi]

(07-15-2014 09:01 PM)Tim Wessman Wrote:  

(07-15-2014 06:13 PM)rprosperi Wrote:  Curious - how did you discover this?

He is Joe Horn. Is any other explanation needed?

Quite right Tim! A silly question indeed.

[rprosperi]

(07-15-2014 10:52 PM)Dave Frederickson Wrote:  Might I suggest "Professor Titan".

Dave

How about "The HP-71 Answer Man" ? Oh, wait. That was taken. A long time ago, in what feels like a Galaxy far, far away. But maybe it's only age?

[Joe Horn]

(07-15-2014 06:13 PM)rprosperi Wrote:  Probably the worst case scenario is packing a volume whose directory has only 1 or 2 open slots. The bogus entries might conceivably get written past the end of the directory table and into the file data area?

Turns out you're exactly right. The INDX89 swap disk (in the MoHPC ftp archive) has 0 free directory entries, which means its file contents begin immediately after the end of the directory. Performing a PACKDIR on INDX89 destroys the contents of the first file on the disk, namely NOTICE71 (a notice from Mark Cracknell, 1 March 1989, forbidding commercial use of the disk) by overwriting its first 256 bytes with garbage. If the file is then copied into an HP-71, attempting to print it can cause a crash.

Furthermore, if PACK is used (instead of PACKDIR) on INDX89, the PACK operation bombs out before finishing, possibly resulting in loss of the entire disk's contents.

Bottom line: Don't use the HP-71's PACK or PACKDIR unless (a) you can see with a hex editor that there is at least one entire free record (256 bytes) between the last directory entry and the beginning of the contents of the first file, and (b) you will NEVER afterwards add anything to the medium using the HP-41 HP-IL module. Otherwise data loss will occur.

[rprosperi]

(08-06-2014 04:16 AM)Joe Horn Wrote:  Bottom line: Don't use the HP-71's PACK or PACKDIR unless (a) you can see with a hex editor that there is at least one entire free record (256 bytes) between the last directory entry and the beginning of the contents of the first file, and (b) you will NEVER afterwards add anything to the medium using the HP-41 HP-IL module. Otherwise data loss will occur.

One thought here. Which version HP-IL Module were you using? I've never found a definitive list of changes from 1A to 1B, but anecdotal comments here and other fora have stated that PACK/PACKDIR were 'better' in 1B than 1A, though they did not go so far as to claim it was completely fixed.

The bug descriptions on J-F's HP-71 bugs page (here) say for item 1184-1 that the PACK problem using a 9114 is due to a 9114 bug, but it seems your findings show otherwise. It is also feasible, but I think unlikely, the issue is within ILPer.

There is also a bug noted in the HP-IL addendum that came with my HP-IL (1A) manual (also noted on J-F's page) about a Memory Lost when using CAT/CAT$ if the first file is deleted, suggesting using PACK to avoid the issue. Definitely a different issue, but sensitivty of first file's proximity to the directory is noteworthy.

[rprosperi]

Some additional info here in a note from Mike Markov on the CHHU05 disk in the SWAP archives, but still can't find the comments I saw earlier.

[Dave Frederickson]

(08-06-2014 06:42 PM)rprosperi Wrote:  Some additional info here in a note from Mike Markov on the CHHU05 disk in the SWAP archives, but still can't find the comments I saw earlier.

This is the same link as in Post #2. The original article is in DataFile V5N3. What comments did you see earlier, Bob?

Dave

[rprosperi]

(08-06-2014 08:21 PM)Dave Frederickson Wrote:  

(08-06-2014 06:42 PM)rprosperi Wrote:  Some additional info here in a note from Mike Markov on the CHHU05 disk in the SWAP archives, but still can't find the comments I saw earlier.

This is the same link as in Post #2. The original article is in DataFile V5N3. What comments did you see earlier, Bob?

Dave

Ooops, sloppy of me to repeat a link, didn't realize that you had posted it above.

When I first got the HPIL:1B module, I went a-googling about "A" vs "B" differences (also noting the issues listed in the errata page that came with my 1A Module Manual) and read somewhere a comment that some aspect of the PACK issues were better using a "B" version, but now can't find it again.

Anyhow, probably best to just avoid PACK-ing at all. With the PIL-Box, tightly managing a few bytes on storage media just no longer makes any sense, and certainly not worth the risk.

[Joe Horn]

(08-06-2014 01:23 PM)rprosperi Wrote:  Which version HP-IL Module were you using? I've never found a definitive list of changes from 1A to 1B, but anecdotal comments here and other fora have stated that PACK/PACKDIR were 'better' in 1B than 1A, though they did not go so far as to claim it was completely fixed.

VER$ reports HPIL:1B. So at least the bugs mentioned in my posts in this thread remain in 1B.