HP-IL checksums or not

[hth]

I have implemented a couple of HP-IL routines to read and write a page to and from 41CL physical memory.

HP uses checksums in their files and an leaning towards doing the same. In that case it would seem most logical to use CRC32 as is done by the YCRC command.

But why are there checksums, and are they needed today?
HP-IL passes messages around and it is strongly recommended practise to verify the data when the frame comes back. Thus, the HP-IL protocol should not introduce any errors. I suppose the reason for checksums was that the magnetic tapes used were not entirely reliable/trusted and detecting a problem would be a good thing to ensure data integrity.

So, is this needed today? If people still use old storage media, then probably yes. For PILBox simulation, maybe not? On the other hand, it would be consistent with existing routines to have a checksum.

So I am pondering over if a CRC32 checksum should be written or not (or something simpler), what do you think?

Håkan

[mfleming]

I think you're right about separating the application domain for checksums between data storage and data transmission. The reliability of modern storage these days is pretty much a given, so storing the checksum with the data may not be entirely justified. That leaves transmission, and since computing the checksum is reasonably cheap each side can exchange their checksum result as part of the transmission acknowledgement protocol.

After all, what use is a checksum if you don't use it?
~Mark

[hth]

Just for the record, I have decided not to store any checksum in the 16-bit ROM files.

According to the VERIFY command:

Code:


************************************************************************
* VERIFY - VERIFY A FILE                                               *
*          WILL ONLY READ THE FILE INTO DRIVER'S BUFFER, NOT TRANSMIT  *
*          THE DATA OUT, BECAUSE ANY PIL TRANSMIT ERROR SHOULD BE      *
*          DETECTED WHILE GENERATING THE FILE.  THE ONLY THING THAT    *
*          MIGHT GO WRONG IS THAT THE DRIVER MAY NOT RECORD ON THE     *
*          TAPE CORRECTLY.                                             *
************************************************************************

It verifies the file by requesting the device to read the file to see that there is data there, no data is actually transmitted.

With modern file systems we can take the storage safety for granted. Should CRCs be needed, I expect it will be applied in the application domain, mainly to pick and match pages for update.

Håkan

[AlexFekken]

Thanks for updating me on the state of modern storage guys.

I will stop wasting my time and money on backups for DR then...

[mfleming]

(11-26-2017 06:06 AM)AlexFekken Wrote:  Thanks for updating me on the state of modern storage guys.

I will stop wasting my time and money on backups for DR then...

Different domains, really.

[Christoph Giesselink]

(11-23-2017 06:31 PM)hth Wrote:  I have implemented a couple of HP-IL routines to read and write a page to and from 41CL physical memory.

HP uses checksums in their files and an leaning towards doing the same. In that case it would seem most logical to use CRC32 as is done by the YCRC command.

But why are there checksums, and are they needed today?
HP-IL passes messages around and it is strongly recommended practise to verify the data when the frame comes back. Thus, the HP-IL protocol should not introduce any errors. I suppose the reason for checksums was that the magnetic tapes used were not entirely reliable/trusted and detecting a problem would be a good thing to ensure data integrity.

So, is this needed today? If people still use old storage media, then probably yes. For PILBox simulation, maybe not? On the other hand, it would be consistent with existing routines to have a checksum.

So I am pondering over if a CRC32 checksum should be written or not (or something simpler), what do you think?

Håkan

Some thoughts about checksums and CRC in connection with HP calculators. I have some experiences with all Saturn based calculators beginning with the HP-71B until the HP-49G.

The cheapest way to detect memory modifications is building a checksum. A checksum is easy to create but has the disadvantage that error detection only works for small memory blocks. Better are CRC's. But CRC needs more CPU power to calculate, moreover the Saturn CPU core don't have the necessary XOR opcode for a fast generation and so this opcode must be rebuild by the AND, OR and NOT operations. This makes it quite hard to write a CRC generator by software. The later Saturnator core in the HP-49G+ and HP50G has the XOR opcode and also of the course the base ARM CPU core.

Because of this, all calculators with the Saturn 1LF2 and 1LK7 CPU use checksums in ROM to verify the proper working of the ROM's during the production process and later in post production for service request.

Calculators with the separate 1LF2 and 1LK7 Saturn are the HP-71B, HP-18C and HP-28C. With the beginning of the Pioneer series, HP used the Saturn CPU only as core component in a multifunction chip containing timers, crystal oscillators, display controller, ...

Therefore HP build HP three chips, the 1LR2 Lewis for the High-End Pioneers with the dot-matrix display, the tinier sister 1LR3 Sacajawea for the Mid-Range Pioneers with the combination of dot-matrix sgements with classic annunciators and finally the 1LU7 Bert for the Low-Range Pioneers with 7 segment display. Because of the larger ROM, HP decided to add a CRC16 hardware generator to the 1LR2 and 1LR3 peripheral. The CRC register in the I/O area is updated on every memory read cycle with the exception reading an address in the I/O area. So HP protected the ROM content now with a CRC instead of a checksum. Only the 1LU7 Bert has no hardware CRC, and so the 10KB of ROM are still protected by checksums.

The next calculator generation, the HP48 with the 1LT8 Clarke and with the Yorke chip had the possibility of data exchange over a serial RS232C interface. For some for me unknown reasons, HP decided to use the Kermit protocol for a secure data exchange. The Kermit protocol needs a CRC16, but different to the one used by the prior Lewis and Sacajawea chip. So the developers of the Clarke chip replaced the generator polynom to the one used by Kermit. The ROM has still CRC's to check the integrity of the ROM content during production and post production for service purpose. Further some RPL objects in RAM contain also a CRC to verify the integrity of the RAM content.

In connection with the HP-49G you sometimes may read something about XMODEM with HP-CRC. XMODEM itself is a small data transfer protocol with less overhead. 128 byte data blocks must be protected by 8bit checksum or a CRC16, the CRC16 is recommended. The CRC16 is mandatory for 1024 byte blocks. But the CRC16 generator polynom of Kermit and XMODEM/YMODEM are different, so HP decided to use the XMODEM protocol but with the Kermit CRC16 generator polynom. So a standard terminal program with XMODEM send/receive possibilties will _not_ work in connection with the Flash-ROM update using the "HP-CRC".

Last rhetorical question, how do you verify data transfers on HP-IL when the controller is listener and one device is talker? The controller is initiating the transfer with a SDA and the talker device respond with the first data byte. In the next step the controller send the received data byte as request for the next data byte until the controller receive an EOT data frame. So the received data is definitely different from the send one, so the controller has no change to verity the received data for correctness. So any verification of data correctness must be done by the data management itself.

Christoph

[hth]

(11-27-2017 05:58 PM)Christoph Giesselink Wrote:  Last rhetorical question, how do you verify data transfers on HP-IL when the controller is listener and one device is talker? The controller is initiating the transfer with a SDA and the talker device respond with the first data byte. In the next step the controller send the received data byte as request for the next data byte until the controller receive an EOT data frame. So the received data is definitely different from the send one, so the controller has no change to verity the received data for correctness. So any verification of data correctness must be done by the data management itself.

Christoph

Thank you for the interesting information!

HP-IL read transfers are verified by the talker (storage device), it sends a DAB (Data Byte) which is passed around the loop, the controller takes the data and passes it on. When it gets back to the talker it checks that it gets back what it sent out, if it is correct it sends the next frame until the ending ETO (end of transmission OK) frame. If a transmission error is detected (it did not get the same DAB back), it will terminate with an ETE frame (end of transmission - error).

Thus, I believe the protocol is reasonable robust as each data frame is checked during transmission.

For the purpose of sending ROMs, I think there should be an application layer that inspects that we got what we wanted. Ideally, it should verify the CRC32 to see that we loaded the intended ROM. I suspect logical errors at this level may be more likely to occur, and as a side-effect it would also verify the data sent.

But feel free to have opinions, I am not stubborn in case people think there ought to be a checksum. It could perhaps even be adaptive in that it could work either with and without a CRC32 checksum in the data file? In that case it could write a CRC32, but allow for reading files either with or without a CRC32.

Håkan

[Christoph Giesselink]

(11-27-2017 07:04 PM)hth Wrote:  

(11-27-2017 05:58 PM)Christoph Giesselink Wrote:  Last rhetorical question, how do you verify data transfers on HP-IL when the controller is listener and one device is talker?

HP-IL read transfers are verified by the talker (storage device), it sends a DAB (Data Byte) which is passed around the loop, the controller takes the data and passes it on. When it gets back to the talker it checks that it gets back what it sent out, if it is correct it sends the next frame until the ending ETO (end of transmission OK) frame. If a transmission error is detected (it did not get the same DAB back), it will terminate with an ETE frame (end of transmission - error).

Good we talk about. I wasn't aware about this. I only known the protocol from the virtual device implementation side and these are imcomplete. JFG's original ILPer implementation and all successors like ILPer 2.x, pyILPer, ILVideo, .... don't contain the talker error checking, they always return the ETO and never the ETE frame at end of transmission.

Even more all virtual controllers, like Emu71/Win (also in device configuration) don't verify what they received in most conditions so they don't recognize transfer errors too.

It should not be too difficult to add ETE in the TALKER implementation of the devices, but the HP-IL simulation inside Emu71 wasn't designed for this.

[mfleming]

(11-27-2017 07:04 PM)hth Wrote:  But feel free to have opinions, I am not stubborn in case people think there ought to be a checksum. It could perhaps even be adaptive in that it could work either with and without a CRC32 checksum in the data file? In that case it could write a CRC32, but allow for reading files either with or without a CRC32.

Håkan

There are a number of user ROMs that lack a valid checksum in the place reserved for one. Angel notes that fact in his own work. Who knows for certain of other non-HP modules. It would be quite a pain for your user if you were to tag the transfer of such a module as faulty.

~Mark

[jsi]

(11-28-2017 02:35 AM)mfleming Wrote:  There are a number of user ROMs that lack a valid checksum in the place reserved for one. Angel notes that fact in his own work. Who knows for certain of other non-HP modules. It would be quite a pain for your user if you were to tag the transfer of such a module as faulty.

The LIFUTILS programs "rom41hx", "rom41er" and "rom41lif" recompute the checksum of the input rom image. If you prepare a rom image as HEPAX-SDATA, ERAMCO-MLDL OS or uncompressed SDATA file in this way you should not worry about missing or wrong rom checksums.

The HEPAX "readrom" command complains checksum errors but stores the rom image into the destination page anyway.

[hth]

We are talking about different checksums here.

The ROM checksum at the last address is not going to be checked by my routines. I am aware that there are ROMs with incorrect checksum and those would be rejected, which is undesirable. They probably should have a correct checksum, but that is another issue.

Also, when saving an arbitrary 41CL 16-bit 4K page for backup purposes, it may be data and not a ROM image, meaning not all images may even have a checksum last.

The checksum I am discussing here is a checksum outside the ROM, that may be stored at the end of the file (after all data image).

Now, I had the impression that HP-IL is transmission safe (within reason) thanks to the intended checking of every frame sent. This is strongly encouraged by HP, but not mandatory. Apparently this has not been the norm for ILPer and friends which means that the idea I based this on cannot be taken for granted.

As we already use CRC-32 as signature for 41CL ROM images, it would make sense to also use that as the checksum here. Why have two ways of calculating some signature/checksum when we can settle for one?

I intend to make the CRC-32 optional, as there may be an application layer that deals with CRC-32 for signature purposes. Besides, the code without checksums is already out, and I cannot just make it incompatible over night.

My plan is to update the CLILUP ROM at some point and have WRROM16 write the CRC-32 checksum after the ROM image. The RDROM16 read function will check the file size and work both with and without a CRC-32 checksums (obviously no checking in the latter case).

Håkan