newRPL - can't restore to 39gs

[franco51]

I use newRPL on a 39gs regularly (it’s great!) but following a crash I want to restore the data.

I’m using a .nrpb file, with 30 USBRESTORE on the 39gs and USBARCHIVE on the virtual calc. When I try to send the file to the physical 39gs I get an “Error Received invalid data” message and the USB link disconnects.

I’ve tried Warmstarts, Resets (physical and software), purging all vars and also going back to the 1089 firmware release and trying again but with no luck.

I can Open this file on the virtual calc and everything runs from there with no problems.

I’m using Win 7 Pro by the way.

What am I missing, please?

BR
Frank.

[Claudio L.]

(03-13-2019 02:19 PM)franco51 Wrote:  I use newRPL on a 39gs regularly (it’s great!) but following a crash I want to restore the data.

I’m using a .nrpb file, with 30 USBRESTORE on the 39gs and USBARCHIVE on the virtual calc. When I try to send the file to the physical 39gs I get an “Error Received invalid data” message and the USB link disconnects.

I’ve tried Warmstarts, Resets (physical and software), purging all vars and also going back to the 1089 firmware release and trying again but with no luck.

I can Open this file on the virtual calc and everything runs from there with no problems.

I’m using Win 7 Pro by the way.

What am I missing, please?

BR
Frank.

Try the following:
Save As... the virtual calculator (which has your data) to a file on the PC and then use the option from the menu to remote RESTORE from a file. This will bypass the simulator completely and send the data file directly to the calc.

Looking at the source code, the USBRESTORE command will only show "Received invalid data" under ONE condition: The received file does not have the letters "NRPB" on the first 4 bytes. This means the data received was likely completely garbled (hasn't happened to me before, comms have been very reliable to me).

Try a different USB port, different cable, different PC. Blow your calculator USB connector with a dust blower, etc. to eliminate the possibility of a bad connection.
Open the file on the PC with a hex editor and double check that the first 4 bytes actually are NRPB (this is for the Save As... method above, where you actually have a file you can check).

Other than that... I don't know. The strange part is that your Desktop version can work with the file without any issues, (so the data itself is good) but the calc isn't even receiving the header correctly.
For the record, the .nrpb file is not just a dump of the objects in memory. Before saving, a there's a pass that converts any object in ROM into special ROMIDs, then objects are saved. Recovery is even more involved: there's one pass reversing the process I just mentioned where all ROMIDs are changed with objects in the new ROM (new because it is potentially completely different, since the file can be restored on a different platform). Then each individual object is checked for validity (same as using the MEMFIX command).

[franco51]

Claudio, many thanks for your reply. The backup /restore process is a lot more complicated than I imagined! The backup file header is correct.
Unfortunately I can't access my Win 7 system until Monday to try anything else. In the meantime I've removed all battery power from the 39gs for a few hours and tried a restore using a Win 10 system ( worth a go because I was getting the same error with both systems). I got a different result this time, from the vertual cac a message stating something like "failed to get a response...“. I'm hopeful that this is just a Win 10 problem now. I'll post the Win 7 results next week.

BR
Frank.

[Claudio L.]

(03-15-2019 10:06 PM)franco51 Wrote:  Claudio, many thanks for your reply. The backup /restore process is a lot more complicated than I imagined! The backup file header is correct.
Unfortunately I can't access my Win 7 system until Monday to try anything else. In the meantime I've removed all battery power from the 39gs for a few hours and tried a restore using a Win 10 system ( worth a go because I was getting the same error with both systems). I got a different result this time, from the vertual cac a message stating something like "failed to get a response...“. I'm hopeful that this is just a Win 10 problem now. I'll post the Win 7 results next week.

BR
Frank.

I wouldn't be so quick to blame Windows, failing to get a response is actually more in line with the assumption of a cable issue or dust in the connectors. I used Windows 10 without issues, but most commonly I do it on a Linux VM running on VirtualBox running on Windows 10, using USB pass through and even with that many things that can go wrong, it still works.

[franco51]

(03-16-2019 06:15 PM)Claudio L. Wrote:  

(03-15-2019 10:06 PM)franco51 Wrote:  Claudio, many thanks for your reply. The backup /restore process is a lot more complicated than I imagined! The backup file header is correct.
Unfortunately I can't access my Win 7 system until Monday to try anything else. In the meantime I've removed all battery power from the 39gs for a few hours and tried a restore using a Win 10 system ( worth a go because I was getting the same error with both systems). I got a different result this time, from the vertual cac a message stating something like "failed to get a response...“. I'm hopeful that this is just a Win 10 problem now. I'll post the Win 7 results next week.

BR
Frank.

I wouldn't be so quick to blame Windows, failing to get a response is actually more in line with the assumption of a cable issue or dust in the connectors. I used Windows 10 without issues, but most commonly I do it on a Linux VM running on VirtualBox running on Windows 10, using USB pass through and even with that many things that can go wrong, it still works.

More testing: As previously suggested the Win version is not significant and clearing the RAM was a red herring; it’s gone back to invalid data messages.

I have tried cleaning the 39gs USB contacts with IPA but as the boot loader always works with the exact same hardware I assume the connectivity is fine. As a double check, I put a UM25C USB Tester in series at the PC end. This class of device is very simple but by putting it in ‘scope mode it does at least confirm that the USB+ and USB- connections are driven by the 39gs USB driver.

I seem to remember something about pinging ponging data so I tried this: { « 1 1 + 2 2 + 3 3 + » } can be shuttled back and forth but I can’t send { « 1 1 + 2 2 + 3 3 + 4 4 + » } or anything longer from virtual to real calculator. The link is disconnected and a zero is returned as before. Going from real to virtual is not a problem with at least twice the length. Forgive me if this has been raised before - I can’t find it.

Any other suggestions, please?
BR
Frank

[3298]

While I have no knowledge about how USB works or how newRPL uses it, a size limit reminds me of VPN affecting UDP traffic by lowering the max size, which can lead to dropped packets because unlike TCP packets the UDP ones cannot be fragmented. Let me throw an idea at the wall and see if it sticks: Conceptually USB hubs and VPN have something in common in that they tunnel traffic ... Are you using a USB hub? If so, try a direct connection.
(The culprit may not actually be a size limit; slow transmission that causes long packets to not be received in entirety before a timeout is over would probably have similar effects. The primary suspects for such bottlenecks are of course devices that literally sit in the middle of the connection, but it might be a driver too since those sit in the software section of the line, so to speak.)

[Claudio L.]

(03-18-2019 12:41 PM)franco51 Wrote:  More testing: As previously suggested the Win version is not significant and clearing the RAM was a red herring; it’s gone back to invalid data messages.

I have tried cleaning the 39gs USB contacts with IPA but as the boot loader always works with the exact same hardware I assume the connectivity is fine. As a double check, I put a UM25C USB Tester in series at the PC end. This class of device is very simple but by putting it in ‘scope mode it does at least confirm that the USB+ and USB- connections are driven by the 39gs USB driver.

I seem to remember something about pinging ponging data so I tried this: { « 1 1 + 2 2 + 3 3 + » } can be shuttled back and forth but I can’t send { « 1 1 + 2 2 + 3 3 + 4 4 + » } or anything longer from virtual to real calculator. The link is disconnected and a zero is returned as before. Going from real to virtual is not a problem with at least twice the length. Forgive me if this has been raised before - I can’t find it.

Any other suggestions, please?
BR
Frank

Now that's really strange, but more specific to help pinpoint the issue. What's failing is the transfer of multi-packet data from virtual to real. That's got to be either a timing issue or some kind of bug. The strange part comes from the code being identical for both virtual and real calc, except for the lower-level layer. I know libusb documentation talks about an extra byte sent by Windows but it's supposed to take care of that internally.

You presented a specific scenario doing { « 1 1 + 2 2 + 3 3 + 4 4 + » } USBSEND from the virtual calc to the real one. That's something we can try to replicate.
Can somebody else confirm this is failing? My calc is dead at the moment (as per the other thread) and until it comes back to life I can't do any testing but perhaps other users can help?
Whether it's a 50g, 39g or 40g, the firmware is the exact same code base so if it fails on one it should fail on all of them.

[Claudio L.]

(03-18-2019 03:49 PM)3298 Wrote:  While I have no knowledge about how USB works or how newRPL uses it, a size limit reminds me of VPN affecting UDP traffic by lowering the max size, which can lead to dropped packets because unlike TCP packets the UDP ones cannot be fragmented. Let me throw an idea at the wall and see if it sticks: Conceptually USB hubs and VPN have something in common in that they tunnel traffic ... Are you using a USB hub? If so, try a direct connection.
(The culprit may not actually be a size limit; slow transmission that causes long packets to not be received in entirety before a timeout is over would probably have similar effects. The primary suspects for such bottlenecks are of course devices that literally sit in the middle of the connection, but it might be a driver too since those sit in the software section of the line, so to speak.)

newRPL uses a standard HID interface, so there's fixed-size packets of 64 bytes. newRPL uses a couple of bytes to identify packet type and send a stream of tiny packets.
The code { « 1 1 + 2 2 + 3 3 + 4 4 + » } compiles to exactly 64 bytes in size, so it would require 2 packets (because a few bytes of overhead on each packet).
But back to the original problem, the first 4 letters "NRPB" are not arriving well, and that's the very first packet. I'm thinking it might have to do with Windows and its extra byte. I don't remember changing the version of libusb I used to build the sources, but perhaps this problem has always been there we just didn't catch it.

EDIT: By the way, using standard HID means there's no driver getting in the way.

[franco51]

(03-18-2019 04:24 PM)Claudio L. Wrote:  

(03-18-2019 03:49 PM)3298 Wrote:  While I have no knowledge about how USB works or how newRPL uses it, a size limit reminds me of VPN affecting UDP traffic by lowering the max size, which can lead to dropped packets because unlike TCP packets the UDP ones cannot be fragmented. Let me throw an idea at the wall and see if it sticks: Conceptually USB hubs and VPN have something in common in that they tunnel traffic ... Are you using a USB hub? If so, try a direct connection.
(The culprit may not actually be a size limit; slow transmission that causes long packets to not be received in entirety before a timeout is over would probably have similar effects. The primary suspects for such bottlenecks are of course devices that literally sit in the middle of the connection, but it might be a driver too since those sit in the software section of the line, so to speak.)

newRPL uses a standard HID interface, so there's fixed-size packets of 64 bytes. newRPL uses a couple of bytes to identify packet type and send a stream of tiny packets.
The code { « 1 1 + 2 2 + 3 3 + 4 4 + » } compiles to exactly 64 bytes in size, so it would require 2 packets (because a few bytes of overhead on each packet).
But back to the original problem, the first 4 letters "NRPB" are not arriving well, and that's the very first packet. I'm thinking it might have to do with Windows and its extra byte. I don't remember changing the version of libusb I used to build the sources, but perhaps this problem has always been there we just didn't catch it.

EDIT: By the way, using standard HID means there's no driver getting in the way.

It would be great if someone else could check this out please.

On my Win 10 system some behaviour is slightly different from my Win 7 system –
n USBRECEIVE on the real calc for { « 1 1 + 2 2 + 3 3 + 4 4 + » } causes an Invalid data notice from the real calc, and the link does not disconnect.

Other results are the same -
{ « 1 1 + 2 2 + 3 3 + » } is received okay on the real calc. Also, it seems that any length is handled correctly when sent to the virtual calc.

The difference in behaviours may suggest timing issues, at least in part.

The other error message I was getting: "Failed to send remote commands to newRPL Calc - serial number - version"

Finally, just to elaborate on the .nrpb file, opening it in Notepad++ I can see NRPB in plain text at the start of the content, followed by NULs, EOTs and so forth. (Notepad++ does see this file as a Normal text file).

BR
Frank

[Claudio L.]

(03-18-2019 11:01 PM)franco51 Wrote:  It would be great if someone else could check this out please.

On my Win 10 system some behaviour is slightly different from my Win 7 system –
n USBRECEIVE on the real calc for { « 1 1 + 2 2 + 3 3 + 4 4 + » } causes an Invalid data notice from the real calc, and the link does not disconnect.

Other results are the same -
{ « 1 1 + 2 2 + 3 3 + » } is received okay on the real calc. Also, it seems that any length is handled correctly when sent to the virtual calc.

This list is less than 60 bytes so it should fit in a single-packet of information. Data sent by the physical calc is prepared by newRPL's USB drivers, so it doesn't have any extra bytes and is properly formatted, I'm not surprised it arrives correctly even for multi-packet files.

(03-18-2019 11:01 PM)franco51 Wrote:  The difference in behaviours may suggest timing issues, at least in part.

The other error message I was getting: "Failed to send remote commands to newRPL Calc - serial number - version"

If for some reason libusb fails to recognize that extra byte and doesn't read it off the stream, all data sent afterwards will be shifted by one byte and all communications will fail. That makes me even more suspect of that extra byte being the issue.

(03-18-2019 11:01 PM)franco51 Wrote:  Finally, just to elaborate on the .nrpb file, opening it in Notepad++ I can see NRPB in plain text at the start of the content, followed by NULs, EOTs and so forth. (Notepad++ does see this file as a Normal text file).

BR
Frank

That's correct and expected. The file is binary and contains compiled RPL objects, but the first 4 bytes are used to identify the file type: NRPB is a full system backup as created by ARCHIVE/RESTORE while NRPL are individual objects (as saved by SDSTO for example).

[Claudio L.]

Following up on this, I took my spare 50g and finally converted to newRPL (after my failed attempts to revive my daily driver). I used the USB port to restore memory from the Desktop version just to try (Linux version, it's what I had in hand at the time). I did USBRESTORE both from within the simulator and doing a save as, then direct USBRESTORE from file. I tried several times and it went without issues every time. I used MKBINDATA to create large chunks of binary data and stored in variables (to make it send many packets over the USB bus) and I could not reproduce your failure.

[franco51]

Claudio, many thanks for following up on this. I’m shelving the matter for now, as no others have reported any similar issues and in the meantime, I am continuing to use newRPL.

A big thanks to Claudio and the team, newRPL has an highly effective UI that makes it so flexible and brilliant to use.

BR
Frank.

[Claudio L.]

One thing you could try is coding a simple program that does VARS then do DUP 1 << RCL >> DOLIST.
Send both lists with USBSEND then write a little program that does << 2 << STO >> DOLIST >> on the target calc. It's annoying to send directories one by one but it might get most of the data to the other side. I wish I could debug what's happening to your calc.

[franco51]

That will work only if a list is not greater than the buffer size - which they are unfortunately.

Actually, you mentioning directories reminded me of the first time I tried the restore function, which was some time ago now. I don't know if it is relevant but the directory structure was lost following that restore. If I recall, all variables were restored okay but as a flat structure. Furthermore, because I had stored some variables with same names in different directories, those were all restored multiple times. I thought I would get rid of some of them but I found that purging one got rid of all with the same name. Since then, I don't use directories; all variables now have unique names relevant to the application.

BR
Frank.

[Claudio L.]

(03-24-2019 10:03 PM)franco51 Wrote:  That will work only if a list is not greater than the buffer size - which they are unfortunately.

USBSEND/USBRECEIVE use one routine. USBARCHIVE/USBRESTORE use a different routine. This means USBSEND might succeed even if USBARCHIVE fails.

(03-24-2019 10:03 PM)franco51 Wrote:  Actually, you mentioning directories reminded me of the first time I tried the restore function, which was some time ago now. I don't know if it is relevant but the directory structure was lost following that restore. If I recall, all variables were restored okay but as a flat structure. Furthermore, because I had stored some variables with same names in different directories, those were all restored multiple times. I thought I would get rid of some of them but I found that purging one got rid of all with the same name. Since then, I don't use directories; all variables now have unique names relevant to the application.

BR
Frank.

Directories are stored flat in memory, with start and end markers indicating where each directory starts. The only reason for those markers to be "lost" is while it was still under development and they may have changed from one version to the other. That hasn't happened in many years. Also, variables with the same name don't get purged all at once. They shadow each other and get purged individually. When you have multiple variables of the same name you need to PURGE multiple times until you get the "Undefined variable" error.
Nowadays there's ZERO reason for you not to use directories. There's no known bugs there, and nothing is actively changing in that area anymore. I haven't lost my directories since 2015, and I crash my development calc all the time, even on purpose.

[franco51]

It gets stranger and stranger. I think the next thing to do is run some hardware checks. I'll reflash Version 2.22 firmware and run all available tests.

BR
Frank

[franco51]

Okay, so the 39gs memory tests didn’t show anything wrong.

I found my old 50g calculator languishing in a draw (it has a broken area on the LCD and is awaiting repair) so I loaded newRPL as an experiment. Unfortunately, the 50g behaves exactly as 39gs when trying to transfer anything more than 64 bytes to it. I was able to restore the 50g data (almost 120 Kbytes) with Conn4x.

I’m going to look into USB Protocol Analysers – something I know nothing about - to see if they could help.

BR
Frank

[Claudio L.]

(03-27-2019 10:20 AM)franco51 Wrote:  Okay, so the 39gs memory tests didn’t show anything wrong.

I found my old 50g calculator languishing in a draw (it has a broken area on the LCD and is awaiting repair) so I loaded newRPL as an experiment. Unfortunately, the 50g behaves exactly as 39gs when trying to transfer anything more than 64 bytes to it. I was able to restore the 50g data (almost 120 Kbytes) with Conn4x.

I’m going to look into USB Protocol Analysers – something I know nothing about - to see if they could help.

BR
Frank

Good, that confirms the hardware of the calc has nothing to odo with it.
I'll have to go fire up a Windows machine and do some testing. I'm curious to see if I can replicate your issues somehow. Is there nobody else using Windows with newRPL?

[bzoom]

Quote:On my Win 10 system some behaviour is slightly different from my Win 7 system –
n USBRECEIVE on the real calc for { « 1 1 + 2 2 + 3 3 + 4 4 + » } causes an Invalid data notice from the real calc, and the link does not disconnect.

Other results are the same -
{ « 1 1 + 2 2 + 3 3 + » } is received okay on the real calc. Also, it seems that any length is handled correctly when sent to the virtual calc.

I have win10, newrpl desktop #1089 and 39gs #1180.
Tested with my setup: Exactly same error as quoted above, short list receives ok but longer one causes invalid data.

[Claudio L.]

(03-27-2019 02:13 PM)bzoom Wrote:  I have win10, newrpl desktop #1089 and 39gs #1180.
Tested with my setup: Exactly same error as quoted above, short list receives ok but longer one causes invalid data.

Thank you. I took the dust out of my old Windows laptop and will begin testing. Something must've broken since I developed the USB part. Also, I'll release an updated newRPL Desktop version as soon as I fix this bug.