[newRPL] Filesystem sanity checks after SD card reinsertion

[matthiaspaul]

Split from the main newRPL thread at http://www.hpmuseum.org/forum/thread-464...l#pid61351

(09-17-2016 06:55 PM)Claudio L. Wrote:  

• Safe notification and exception when card is removed with open files
  

I like this a lot and have a suggestion how to further improve it:

At present there appears to be no protection if the user would reinsert a different card or change the contents of the card externally before reinserting it.
So, the system could store some signature identifying the card alongside the other card data. If the card gets removed before all data has been written out and the system detects that a card is inserted again, it would compare the card's signature with the internally stored one and write out the pending data only if the card was identified to be the same card as before. If, according to the signature,
it isn't the same card, the user is prompted to insert the correct card or discard the pending data.

If the medium has a DOS 3.4 EBPB (with signature 28h at sector offset +026h / BPB offset +1Bh) or a DOS 4.0 EBPB (with signature 29h at sector offset +026h / BPB offset +1Bh), a 32-bit volume ID is stored at sector offset +027h / at +1Ch in the BPB.
If the medium has a DOS 7.1 EBPB (with signature 28h or 29h at sector offset +042h / BPB offset +37h), the 32-bit volume ID is instead stored at sector offset +043h / BPB offset +38h.
This ID is changed whenever the medium gets formatted and it can be used to identify the medium.
While this ID exists on most media today, some embedded systems do not properly write it. Also, it may not be set to individual values on some mass-produced media. Sometimes, older BPB formats are used even by modern formatters (sometimes only under certain conditions) - the volume ID is not present in any BPB format before the 3.4 format, and f.e. the DOS 3.31 BPB format is still in use today.
Since it is technically possibly to use a DOS 7.1 EBPB also for FAT12 and FAT16 volumes as well as a DOS 4.0 EBPB for small FAT32 volumes (with some restrictions), the BPB format must be detected independent of the FAT filesystem type. There are many subtle details to be considered, so this will be subject of another thread in the future. However, for the purpose of just checking signatures it isn't really necessary to determine the actual BPB format, you could just store away and compare the 32-bit values stored at +1Bh and +38h. If they would contain other data in other BPB formats, this would eliminate the extra safety net created by comparing signatures, but the code would continue to function.

While the volume ID above is the most important signature to be checked, there are a number of additional signatures which could be checked in the same way:

You could additionally check the 8 bytes at sector offset +003h, called the OEM label. Some vendors use this entry to store licensing keys.
However, there's a problem here: The Volume Tracker in Windows 9x stores special tracking IDs here (and this happens even on read-only operations like "DIR A:" unless the medium is write-protected). This invalidates the check. In order to avoid the problem, including the OEM label in an extended signature check is valid only if the 3 bytes at sector offset +00Ah do not read "IHC".

Similar, on partitioned media, there are two 32-bit values at +0DCh (disk timestamp) and +1B8h (disk signature) in the MBR which could be used in an extended signature check as well. They do not exist in all MBRs, but their presence can be detected by special magic values in nearby bytes. For the purpose of checking for media identity as part of an extended signature check, however, these extra validity checks are unnecessary.

Finally, the SD card itself has a 32-bit product serial number (PSN) stored in bits 55-24 of its CID. I don't know, if all cards support this, though.

So much for more or less static signatures.

Some additional sanity tests:

On media with FS info sector, the last cluster and free counters could be used as an additional sanity check. If these would have changed from the last internally stored values, the medium must have been written to externally. Depending on what kind of data still needs to be written out it may or may no longer be safe to do so, even if the medium could be identified as the correct one. In this situation the original scenario cannot be reconstructed and the calculator could skip prompting the user and just discard the pending data with an error message.

In a similar way, the calculator could check the last modified time of previously open files. If their timestamp has changed externally, any still pending data relating to these files could be discarded with an error message as well.

Greetings,

Matthias

[Claudio L.]

(09-18-2016 08:43 PM)matthiaspaul Wrote:  Split from the main newRPL thread at http://www.hpmuseum.org/forum/thread-464...l#pid61351

(09-17-2016 06:55 PM)Claudio L. Wrote:  

• Safe notification and exception when card is removed with open files
  

I like this a lot and have a suggestion how to further improve it:

At present there appears to be no protection if the user would reinsert a different card or change the contents of the card externally before reinserting it.
So, the system could store some signature identifying the card alongside the other card data. If the card gets removed before all data has been written out and the system detects that a card is inserted again, it would compare the card's signature with the internally stored one and write out the pending data only if the card was identified to be the same card as before. If, according to the signature,
it isn't the same card, the user is prompted to insert the correct card or discard the pending data.

It's done, but there's a weakness I haven't been able to resolve. Right now I'm using the card CID which is unique to each card to make sure the user inserts the right card. The exception warns the user and asks to reinsert the card immediately, so no data is lost. If the user doesn't care and inserts a different card, the system detects the card was changed and does a special shutdown of the file system that discards everything without writing it to the card. The new card will remain unmounted until first use.
In that regards the protection works as you intended.
What I can't prevent is the user from pulling the card with a dirty file system, carelessly disregard the request to reinsert the card, then use the card on a PC, which would make changes to the file system, and then reinsert the card on the 50g. The 50g would detect the same card is back and would keep the volume mounted, which risks corrupting the FAT if we had any pending writes to the FAT.
This case is quite rare, though, because the user has to pull the card within 3 seconds of the last write, but it could happen and I don't have a solution in place for it.

The last cluster and free space only work on FAT32, so that doesn't help. Also, moving a file to another directory may not allocate any clusters, keeping those counts the same. However, the altered directory may have open files on the 50g, which means there could be up to one sector of buffered data ready to overwrite whatever the PC did.

Any other way to achieve this?