HP 35s Checksum explained

[Tugdual]

I saw a number of discussions about the 35s check sum calculation so I decided to have a look on it and could figure out the calculation method that I'm now going to explain here. Unfortunately I am not going to tell you why it returns different values on different calculators, I don't know myself and as a matter of fact the simulator and the physical calculator are pretty consistent for me.

Program storage
Each instruction is stored in a block of 3 bytes. The indication LN=xxx is the exact number of bytes consumed by your application. If you divide by three it should return the number of lines. I could decipher most of the OP codes and this 3 bytes size seems to be fairly constant.

On the simulator, if you save the state and check the produced EP file, the programs are stored at the very end of the file, in reverse order (last 3 bytes are the 1st op code).

Magic values
The check sum algorithm is using 6 magic values:
M1:1021h
M2:1231h
M3:3331h
M4:0373h
M5:3730h
M6:4363h

Now let's see how these values are used to calculate the check sum of a very simple program.

Code:

PRGM TOP
A001 LBL A

Now both my calculator and the simulator tell me

Code:

LBL A
LN=3
CK=CBFD

Hexa dump
If you save the state of the simulator and check the end of the file you will find:

Code:

01 01 7B 3A 00

01 is apparently a multi purposes bitset of flags
01 is a parameter, here it means 'A'. LBL B would be 01 02 7B
7B is the op code of LBL
3A 00 you can ignore, this is a constant marker for the end of the file.

Let's translate this into 6 binary nibbles

Code:


0    1    0    1    7    B
0000 0001 0000 0001 0111 1011

Now we associate a magic number to each nibble in the reverse order

Code:


M6   M5   M4   M3   M2   M1
0000 0001 0000 0001 0111 1011

Each nibble has four bits b0, b1, b2, b3
For nibble 1, we calculate V1 = (b0*M1) XOR (2*b1*M1) XOR (4*b2*M1) XOR (8*b3*M1)
V1 = 1021h xor 2*1021h xor 8*1021h = B16Bh
V2 = 1231h xor 2*1231h xor 4*1231h = 7E97h
V3 = 3331h
V4 = 0
V5 = 3730h
V6 = 0

Now we calculate
V1 xor V2 xor V3 xor V4 xor V5 xor V6
CFFCh xor 3331h xor 3730h = CBFDh

Composing multiple instructions
Let's add an instruction after the label

Code:

PRGM TOP
A001 LBL A
A002 SIN

LBL A
LN=6
CK=1B94

Now the hex dump looks like this:

Code:

01 00 43 SIN
01 01 7B LBL A

Using the same method, we find that the SIN instruction produces a checksum of 4F97h.
We sum this with the value found previously for LBL A
4F97h + CBFDh = 11B94h

The 35s is dropping values above 4 digits and this is how we achieve 1B94h

Conclusion
The calculation doesn't yet include strings. The 35s stores EQN is a specific separate memory area while the opcode has a pointer (offset) and I haven't yet checked how they were included; may be this could be the problem because the pointer is part of the opcode and entering EQNs in random order stores strings in random order. I will verify this later.
Apart from that, the calculation is robust and deterministic and so far I always achieved the same results in the simulator and on the physical calculator.

[Tugdual]

Topic on litterals
As I mentioned before, when entering a EQN, a string (in ASCII) is stored in a different location.
The OP CODE remains 3 bytes long but the count of # of characters is added to the total.

Few remarkable aspects
- values entered in the code are stored the same way as EQN though they don't have the same OP CODE. Therefore they share the same memory area.
- each string is stored in a 35 bytes block. The 2 first bytes indicate the length. It seems to be possible top have larger strings and in that case the 35s stores them in multiple blocks, not sure how.

Entering

Code:

A001 LBL A
A002 1

and

Code:

A001 LBL A
A001 2
A001 1

Then removed line A001 produce the same code but the string are arranged differently and the pointer in the second case is 35 bytes farther than in the first case. Still the CRC is the same indicating that the pointer valued is not used to calculate the CRC.

Conclusion
Litterals (strings entered with EQN or constant values) seem to be handled properly in the CRC calculation. At the moment I still don't know why some calculators should return different values. I also cannot reproduce the problem and my hardware and simulator are always returning identical values.
I would be curious to see if some people get different values on their simulator. We're not sure that all calculators are strictly identical (same ROM? Some are may be also defective) but if the simulator returns different values in some occasions this would definitely indicate that we have a memory context that impacts the calculation.

[David Hayden]

Awesome! Thanks for working this out. I suspect the problem is that the bytes in a program are not position independent. In other words, the same program might contain different bytes depending on where it's located in memory. This would be true if, for example, a goto contains the address of the target instruction.

[Gerald H]

(02-09-2015 09:20 AM)Tugdual Wrote:  Topic on litterals
As I mentioned before, when entering a EQN, a string (in ASCII) is stored in a different location.
The OP CODE remains 3 bytes long but the count of # of characters is added to the total.

Few remarkable aspects
- values entered in the code are stored the same way as EQN though they don't have the same OP CODE. Therefore they share the same memory area.
- each string is stored in a 35 bytes block. The 2 first bytes indicate the length. It seems to be possible top have larger strings and in that case the 35s stores them in multiple blocks, not sure how.

Entering

Code:

A001 LBL A
A002 1

and

Code:

A001 LBL A
A001 2
A001 1

Then removed line A001 produce the same code but the string are arranged differently and the pointer in the second case is 35 bytes farther than in the first case. Still the CRC is the same indicating that the pointer valued is not used to calculate the CRC.

Conclusion
Litterals (strings entered with EQN or constant values) seem to be handled properly in the CRC calculation. At the moment I still don't know why some calculators should return different values. I also cannot reproduce the problem and my hardware and simulator are always returning identical values.
I would be curious to see if some people get different values on their simulator. We're not sure that all calculators are strictly identical (same ROM? Some are may be also defective) but if the simulator returns different values in some occasions this would definitely indicate that we have a memory context that impacts the calculation.

Given the complexity & correctness of all your good work, what is the utility of these "check sums"?

[Didier Lachieze]

I've tested the small program listed in this thread from 2007: 35s checksum problem

On my real 35s (ROM version 1 - CNA743xxxxx) I have:
LBL G, LN= 9, CK= 535D
LBL C, LN= 9, CK= 532C

On the 35s emulator (Version 2012 12 10 Rev:19048) I have:
LBL G, LN= 9, CK= B556
LBL C, LN= 9, CK= 7C6B

Both real and emulated 35s had already some (different) programs inside before the test. On both I first did GTO .. before entering the program.

[Tugdual]

(02-09-2015 01:01 PM)Didier Lachieze Wrote:  I've tested the small program listed in this thread from 2007: 35s checksum problem

On my real 35s (ROM version 1 - CNA743xxxxx) I have:
LBL G, LN= 9, CK= 535D
LBL C, LN= 9, CK= 532C

On the 35s emulator (Version 2012 12 10 Rev:19048) I have:
LBL G, LN= 9, CK= B556
LBL C, LN= 9, CK= 7C6B

Both real and emulated 35s had already some (different) programs inside before the test. On both I first did GTO .. before entering the program.

This is pretty interesting.
I have :
G 8A28
C D6F8
on both the simulator and the calculator. I see on the other thread that many other users achieved the same result which seems to indicate some consistency.

I have the exact same version of simulator and my 35s is a CNAxxx version. It would be interesting to start from a common state to understand what's going on.
Can you try to delete AutoSave.ep from %AppData%\Hewlett-Packard\hp35s and then run the test again?
For the hardware version, could be a good idea to remove both batteries to force a complete reset.

[Didier Lachieze]

(02-09-2015 04:07 PM)Tugdual Wrote:  Can you try to delete AutoSave.ep from %AppData%\Hewlett-Packard\hp35s and then run the test again?

I've renamed AutoSave.ep to AutoSave.ep.save and when I launched the emulator I got a MEMORY CLEAR message. Then I got the same checksum values as you.

(02-09-2015 04:07 PM)Tugdual Wrote:  For the hardware version, could be a good idea to remove both batteries to force a complete reset.

I prefer not to erase the programs I currently have in my 35s (don't want to enter them again.. ).
Anyway the point here, as was established soon after the launch of the 35s, is that the checksum values on the 35s are not reliable as they depend on the calculator context and not only on the program for which the checksum is provided.

[Tugdual]

(02-09-2015 04:37 PM)Didier Lachieze Wrote:  

(02-09-2015 04:07 PM)Tugdual Wrote:  Can you try to delete AutoSave.ep from %AppData%\Hewlett-Packard\hp35s and then run the test again?

I've renamed AutoSave.ep to AutoSave.ep.save and when I launched the emulator I got a MEMORY CLEAR message. Then I got the same checksum values as you.

Interesting.
Can you somehow attach the "old" EP file you were using? I'd like to review it...

Note: it is very easy to corrupt a EP file... I've done it a lot to achieve my "findings". We'll never know if that is a ROM bug or a PC issue at the first place unless we have a very deterministic way to reproduce it starting from "MEMORY CLEAR" state.

[Didier Lachieze]

(02-09-2015 04:42 PM)Tugdual Wrote:  Can you somehow attach the "old" EP file you were using? I'd like to review it...

  AutoSave.zip (Size: 1.82 KB / Downloads: 4)

[MarkHaysHarris777]

(02-09-2015 04:37 PM)Didier Lachieze Wrote:  Anyway the point here, as was established soon after the launch of the 35s, is that the checksum values on the 35s are not reliable as they depend on the calculator context and not only on the program for which the checksum is provided.

I don't believe it was established, not in a deterministic way. Also, there is a difference between 'reliable' within context, and 'reliable' for context. If two different machines produce two different checksums (and I cannot reproduce that on my machines so far) then the checksum might NOT be reliable for contextual checking (between machines), but might be absolutely reliable within the context of the machine itself.

[Tugdual]

Unfortunately I have a bad news... I just found that there is indeed a major flaw with this checksum calculation.
Literals, i.e. any string in your code including EQN as well as constant values are sharing a certain memory area and therefore are allocated by blocks. The 3 bytes OP Code contains an offset to its literal ASCII string which seems to be included in the checsum calculation. And this is really a very stupid design because the checksum will depend on your calculator context.
In particular all EQN are sharing, again the same area as any literal in your code. Therefore it is pretty easy to demonstrate the problem, starting from a MEMORY CLEAR state:
Case 1

Code:

[MEMORY CLEAR]
LBL A
1
->7991

Case 2

Code:

[MEMORY CLEAR]
EQN 123456789 // first create a random EQN before you start editing your code
// Now enter the same code as before
LBL A
1
->7A1D

In the case 2 the literal "1" doesn't have the same address as in case 1 because it is pushed ahead by the previous EQN declaration and this is fooling the CRC calculation.

Conclusion
The idea of implementing a CRC was very good, the way it is done makes me feel extremely sad for myself as a customer and for Hp. How sbdy can spend so much effort designing a CRC procedure and then fail so miserably on such an obvious and basic aspect is above me. I'm afraid this definitely concludes the question of CRC.

This being said, I keep thinking that this calculator is a good one despite its flaws and most of the calculations I did to establish these findings were done using the 35s Hexa capabilities, (except some brute force for which I used the PC... much much faster). Special thanks to Didier who somehow helped me to understand what was going on.

[MarkHaysHarris777]

(02-09-2015 10:36 PM)Tugdual Wrote:  In the case 2 the literal "1" doesn't have the same address as in case 1 because it is pushed ahead by the previous EQN declaration and this is fooling the CRC calculation.

Except for one small problem... I deleted *all* of my test equations and *none* of my programs with literals changed... nor did their checksums.

I then added a random EQN as you did
EQN 123456789

... and none of my programs with literals changed, nor did their checksums. In fact, since I have been on this journey I have been adding equations and programs and *none* of the checksums have changed (I have not tried any of this on the emulator).

Something else must be involved (or the hardware itself). Later tonight I'll start with memory clear on one machine and try to reproduce what you have described... I'll get back to you.

marcus

[Tugdual]

(02-10-2015 12:16 AM)MarkHaysHarris777 Wrote:  Except for one small problem... I deleted *all* of my test equations and *none* of my programs with literals changed... nor did their checksums.

I then added a random EQN as you did
EQN 123456789

... and none of my programs with literals changed, nor did their checksums. In fact, since I have been on this journey I have been adding equations and programs and *none* of the checksums have changed (I have not tried any of this on the emulator).

Marcus try this: call Mem and check the amount
Create one more short EQN and check again Mem
You should see that 35 bytes were consumed.

This is exactly what I observe in memory. Any EQN or literal is using a block of 35 bytes. Now if you enter a very long EQN or literal, multiple blocks will be consumed and the memory management looks like a chained list of blocks (with offset of previous, next items but this is a bit unclear to me).

Once a block is allocated, it has no reason to move... so even if you clear EQN all the blocks allocated for your code literals will remain at the same place. This is why it is important to start from a known state (MEMORY CLEAR) and execute a script to predict what happens in the memory organization.

Now I have seen some blocks moving in some occasions but I cannot explain when and why. May be a CLEAR is sometime calling for some basic internal reorganization to avoid too much memory fragmentation. I'm not sure about that.

Also I have not been able to figure out how exactly the checksum of EQN and literals is calculated. I'm not far but there is always a random difference of 2 or 3 that I cannot explain. I also couldn't figure out all the details of blocks management though I have a rough idea.

Anyway for now, my conclusion is that I'm certain about checksum issue and so far I could:
- reproduce the error starting from a known state
- get a basic understanding of why we have the error (totally design flaw, shame on Hp)

Having more understanding would be worthless since there is no way to actually fix it and also my point was to verify that this was not related to hardware issues or code bug. In a sense you can think positive: the issue is not at all random or buggy, it is totally deterministic and totally a design flaw.

[MarkHaysHarris777]

(02-09-2015 10:36 PM)Tugdual Wrote:  Case 1

Code:

[MEMORY CLEAR]
LBL A
1
->7991

Confirmed

Quote:Case 2

Code:

[MEMORY CLEAR]
EQN 123456789 // first create a random EQN before you start editing your code
// Now enter the same code as before
LBL A
1
->7A1D

Confirmed


Thank you, this is the first time I'm getting closer on this. Now I have a catch for you to try. I have also determined that the checksum for (programs, not equations) is based on whether there are equations in memory *first* or not. If equations exist in memory *first* then the A program gets checksum 7A1D (You'll notice our two machines are consistent) but if equations are placed in 2nd (programs are first) then the A program gets checksum 7991 (also consistent with your machine.

BUT... once that *firstness* is established adding programs and equations to memory does NOT change any of the checksums. In other words, once the literal program A is in memory I can add equations with literals all day long without changing A's checksum. Also, on the other hand, if the equation (123456789) is in memory *first* and program A gets checksum 7A1D I can add additional programs or equations into memory without changing any of the checksums.

Of course, I must agree, there is a dumb design issue here because the checksum for the programs (nor equations) should be based on position in memory! CONCLUSION: yes, the checksums are only useful "for context" (between machines) IFF we know how the primary context was established (at any rate this explains why checksums are very consistent "within" context because once the order of memory usage has been established the checksums are not going to change---the reason why I have not been able to reproduce this up until now... so again, thank you!

I must add, for me personally, on the HP35s the most important use of checksums is for confidence 'myself' (to know whether my own context has *changed*). If I modify a program by altering a GTO or XEQ target (say against my own utility library) I want to be alerted to the fact that 'renumbering' may have occurred. I may use the checksums to get that confidence. On the other hand, cross checking between machines (I have two of them) is only going to be relevant (to me, on my units) if I setup the machines the same way (programs first, equations second). Unfortunately, based on this new information, I am not able to have that same confidence between my unit and someone else's (yours perhaps) since I have no idea whether you had equations in memory before you entered any programs!

Again, I plan to run the wheels off this thing. I think its worth having (obviously, I own two of them) and I think its worth using professionally. But we do need to keep the caveats in mind, and we do need to understand the limits of operation.

PS I pulled out my TI30 yesterday (original) and put it into a tight loop just for fun... if you entered this on the 1976 TI30 0 INV TAN that silly thing would go into a tight loop and not come out until you powered off, and sometimes not till you removed the 9v battery! TI fixed that in the later models (my July 1977 model, for instance, does not have the quirk). My point for bringing it up is that when we know what the 'quirk' is, and how to avoid it, then we really don't have any problem.

PSS Nice to meet you, and thanks for working with me on this... it has been most helpful!

Cheers,
marcus

[Thomas Radtke]

(02-11-2015 04:08 PM)MarkHaysHarris777 Wrote:  [...] I think its worth using professionally.

Same here, but that's more a matter of personal preference for this kind of calculators and the wish to have all those new functionality in comparison to its predecessors. It would be much more reasonable to use a 42S, 32S(II) or a 15C imo as I think finding new bugs is more likely with the 35s. I hope I can estimate the results well enough to catch a new problem.

[MarkHaysHarris777]

(02-11-2015 04:37 PM)Thomas Radtke Wrote:  

(02-11-2015 04:08 PM)MarkHaysHarris777 Wrote:  [...] I think its worth using professionally.

Same here, but that's more a matter of personal preference for this kind of calculators and the wish to have all those new functionality in comparison to its predecessors. It would be much more reasonable to use a 42S, 32S(II) or a 15C imo as I think finding new bugs is more likely with the 35s. I hope I can estimate the results well enough to catch a new problem.

Yes, but, the 42s is still way too expensive ( I don't have one yet, and I'm not likely to have one until it can be had for less that $100.00. I'm noticing they're still selling on E-bay for anywhere between $250 and $320 dollars. On the other hand, if you have a 34s you *have* a 42s and a 16c ! The 35s and the 34s are in two separate classes... that 34s is in a league of its own/ truly phenomenal!

[Thomas Radtke]

I'd have one if it weren't for those stickers. OTOH I don't need much beyond what my 32SII, which I use at home, can do.

The 35s in my office has the advantage of being capable to store *many* equations and programs, and still can integrate :-). And if it gets stolen it can easily be replaced.

HP: Develop a 35SII with open firmware!

[MarkHaysHarris777]

(02-11-2015 05:16 PM)Thomas Radtke Wrote:  I'd have one if it weren't for those stickers. OTOH I don't need much beyond what my 32SII, which I use at home, can do.

I thought the same thing (really bugged me actually) until I made my own 34s. The overlay actually was easy to install (just follow Eric's video, lay down the middle and let the ends fall away) and it only took me about 20 minutes to put on all the stickers. BUT, the vinyl stickers are a *very* nice quality and the adhesive is superb. Use clean dry hands (no oily finger tips!) And actually, the 34s really looks good too. Yes, if it had double moulded plastic keys from the factory things would be better, but, all in all its very nice; no complaints, really.

Quote:The 35s in my office has the advantage of being capable to store *many* equations and programs, and still can integrate :-). And if it gets stolen it can easily be replaced.

True enough. That's really the reason I purchased the 35s... it has a ton of memory, its easy to program, and its memory management is easy and flexible (not to mention huge).

Quote:HP: Develop a 35SII with open firmware!

Yes. I think the community is going to beat them to it (43s). In my opinion the machine needs to be Armed, have an all-points-addressable lcd which will allow for four lines (stack levels) with annunciators (allowing graphing if desired) comm built in (peer to peer, and pc, 2.5mm i/o and micro usb) rocker click keys (with cursor keys) and NO larger than the HP35s (its boarder line too big). Obviously, it needs to be flash-able. Building a calculator in this age without being able to replace its firmware (should there be quirks, and you know there will be) is just plain dumb engineering; sorry, IMHO.

I have some ideas about the actual hardware design, which I will post after I get organized, to the 43s status page. I'm almost tempted to build my own prototype... I'm getting restless waiting .

Cheers

marcus

[Tim Wessman]

Quote:Building a calculator in this age without being able to replace its firmware (should there be quirks, and you know there will be) is just plain dumb engineering; sorry, IMHO.

I think you meant to say "enabling unauthorized modification to support cheating in exams"...

(which is how all non engineering types view things unfortunately)

[Tugdual]

(02-11-2015 06:39 PM)Tim Wessman Wrote:  

Quote:Building a calculator in this age without being able to replace its firmware (should there be quirks, and you know there will be) is just plain dumb engineering; sorry, IMHO.

I think you meant to say "enabling unauthorized modification to support cheating in exams"...

(which is how all non engineering types view things unfortunately)

Tim, please don't forget that many people but students use calculators. I really think that a HP design with an open firmware is a new concept that has never been done and would find a public of professional users. I am under the impression that HP is now totally and exclusively focusing on the school market.