newRPL: [UPDATED April 27-2017] Firmware for testing available for download

[Claudio L.]

(07-27-2016 08:56 PM)matthiaspaul Wrote:  That's right, these values are nothing that can be relied upon, they can be used only with sanity checks in place. However, its still possible to take advantage of them.

The idea behind the initial scan was to find the largest pool of free clusters, and keep not only the cluster, but how many we have available until we need to scan for the next hole.
Being the largest hole in the disk, unless the disk is filled near capacity, or fragmented beyond hope, it's unlikely you'll run out of them. This means you only need to read the FAT table at mount, and hopefully never again for a long time.

(07-27-2016 08:56 PM)matthiaspaul Wrote:  

Quote:b) Implementing it requires proper mount/unmount, which means if the user just pulls the card out then putting on a PC will indicate a "dirty" file system and will suggest (or force) disk checking procedures.

Yes, but this is exactly what should happen in this scenario, as the integrity of the file system cannot be trusted any more until after running a disk check utility.

Yes, and no.
When you open a file, the FAT table is read to get the complete chain of clusters for that file. After that, it's only written when you close the file.
Writing to the FAT is done using a "patching" cache. It keeps only a list of clusters that need to be modified, and if anybody reads the FAT table, the sectors read get patched "on the fly" with the contents of the cache, so the process sees the "new" FAT table, even though it wasn't written to disk yet.
Only when you close a file its directory entry gets updated and the FAT cache gets flushed. This also means if you pull the card, the file system is normally in a consistent state (as long as you don't pull it during a write operation).
Well, this is usually the case but it isn't guaranteed (the FAT cache is limited size, so if you write a large amount of data, it will have to perform partial flushes), so there might be lost clusters worst case, but other than that it's safe to pull the card (of course any open files would not be written).
Only if the card is pulled while actually writing there could be an inconsistent state.
And since it almost always is in a consistent state, I tried to avoid writing the boot record to change the dirty bit, because either I had to write it every time I closed a file, or you'd get bothered with a dirty state when the file system is perfectly fine.

(07-27-2016 08:56 PM)matthiaspaul Wrote:  At least this is what I would propose as it is easy to implement (almost no memory and code overhead) and it can speed up things considerably if the values are (almost) valid, and does not cause actual problems, if they are not.

It would speed up mounting, but slow down actual use, as the FAT table would have to be read more often than with the current implementation. In other words, it would spread out those initial 7 seconds into every single write operation.
Still, an idea worth considering, especially now with larger SDHC cards.

(07-27-2016 08:56 PM)matthiaspaul Wrote:  Of course, there are other methods to speed up certain access patterns and there are various strategies how to possibly reduce fragmentation on FAT file systems (the above method is part of what is used by DOS and Windows). Unfortunately, they require considerably more complex implementations, more memory for various types of buffers and to hold dynamically built in-memory data structures, background processes - way too complicated for an embedded system, IMHO.

Yes, I don't want to over complicate it. This is not a Disk Operating System. The file system is just a module of the calculator, the main goal is the calculator, not disk management.
newRPL's file system is not dumb by any means, it is quite optimized for the little resources it uses.

(07-27-2016 08:56 PM)matthiaspaul Wrote:  One feature may be worth considering, though: A vast amount of fragmentation is caused by frequent allocations and deallocations of files, as the system would try to maintain the integrity even of void data (to allow later undeletion), and it would thereby effectively cause more fragmentation in this scenario. However, a good amount of such interim file operations could be carried out on temporary files. Therefore, some operating systems (including DOS) have special API functions for temporary files. They do not only ensure the creation of unique file names (so the user does not have to be bothered with them), but using these functions will also cause the file system to use different allocation / deallocation strategies. The file system would no longer attempt to maintain deleted directory entries and use "fresh" clusters for new allocations, but it would try to reuse previously freed entries.

Something like this could be implemented in newRPL as well. On the command line there could be a number of "reserved file names" which the system would recognize as temporary files. The on-disk file names could use a special pattern so that the system can recognize them as temporary files (even if they are left-overs from previous sessions). The file system could thereby automatically remove orphanted temporary files.

I'm not sure the use case in newRPL would require so many temp files. That's more for a general purpose operating system. newRPL use case is more of writing lots of tiny 1-cluster files. This virtually eliminates fragmentation, as files are (almost all) atomic.

(07-27-2016 08:56 PM)matthiaspaul Wrote:  Yes, I know, there are also a number of other special cases:

- If a filename fits into the 8.3 format with all characters uppercase, Windows can be configured to only create a SFN and skip creating the unnecessary LFN (thereby avoiding unnecessary clutter in the filesystem).

- If a filename fits into the 8.3 scheme and either contains only lowercase letters or combines a lowercase filename and an uppercase extension or vice versa, the creation of an LFN can be suppressed as well. In this case only an SFN is created and the case information is stored in bits 4 and 3 at offset 0x0C in directory entries, so that the LFN can be recreated from the SFN later on.

- Further, Windows can be configured to not start using numeric tails until actually necessary. It would simply truncate the name to fit into the 8.3 scheme, so the SFN for a file named "helloworld.txt" would be "HELLOWOR.TXT", not "HELLOW~1.TXT". Useful to keep as much of the original name available as SFN.

All cases above are implemented exactly as described in newRPL.

(07-27-2016 08:56 PM)matthiaspaul Wrote:  

Quote:None of the above. I just can't picture a multi-user calculator... how do you place more than 2 fingers on the same keyboard? :-)

That's why I wrote "single-user permissions".

While I have no intention of going above and beyond basic VFAT support for newRPL, I'll keep these ideas in mind if I ever decide to improve my CleanFS project.