libhpcalcs: portable (Windows / MacOS X / Linux) connectivity kit library

[debrouxl]

Hmm. This logging output shows that the data received by libhpcalcs is pure garbage:
* the "unknown marker at beginning of image" message;
* the last packet is entirely made of 0x00, which never happens with a valid screenshot;
* last but not least, the maximum expected size for a type 8 screenshot (320x240x16 bpp) is 153600 uncompressed bytes, i.e. quite a bit less than the 199395 bytes declared by the packet. I'm not aware that zlib compression has the property of blowing up the size of non-compressible output by as much as 20+%.
Type 8 screenshots are typically 18000-20000 bytes for a screenshot of the Apps screen, fewer bytes for other more homogeneous screens (typically 6000-7000 for an empty Home / Function screen).

Granted, some of the cleanups which I pushed this morning did change some code paths exercised by attempting to obtain a screenshot:
* "Extract function for computing the size of a Prime virtual packet, export it from the library." - but it can't be that one, the change is a no-op;
* "Make the "data" argument of cable receive functions a double pointer; use that to optimize several functions in prime_cmd." - both the rpkt and the link_prime_hid layers were changed.
However... the exact same relevant code paths were exercised when you received a file, which was a success !

To sum up: I don't know why this happens...

By chance, does the problem go away after simply rebooting your calculator ?

[LarsF]

I tried to reboot both calculator and computer, but no difference, and I try to connect i to monitor and the same behavior occurs but it's much slower when i connect it trough the monitor, both on receive files and screenshots!! I don't have a real USB hub, but I will try to get one and test it with that.

[debrouxl]

libhpcalcs now builds on FreeBSD 10.0 RC4, after a fix suggested by Tijl Coosemans. Tijl is a contributor to many open source projects for *BSD support and portability fixes. I've occasionally been in touch with him for libti*, and now for libhpcalcs.
hidapi does not, however, build on FreeBSD 10.0 RC4 out of the box.

[Egan Ford]

While I am not having problem with large downloads or small uploads, large uploads from OS/X to the Prime are very troublesome.

While developing Breakout I starting having upload issues around 20K in size. There was no indication on either side of a failed upload except that the file was never updated. If I delete the file, still nothing. After repeated power on/off events I could eventually get an upload. A good sign that an upload worked is that I get kicked out of the program listing view back to home view.

If the calculator is cold (i.e. off for a long time), then the first large upload always works.

Twice I had corrupted uploads.

[debrouxl]

Could you provide the full terminal output, presumably through tee, of a failed upload and a successful upload ?

Like the protocol used by TI-Z80 and TI-68k calculators, the protocol used by the Prime is not robust against packet losses "on the wire" (in the USB HID stack, mostly). There's no retransmit scheme. Unlike the computer, the calculator does not increment the first byte in the packet (at least, it didn't in the firmware from August), so calc -> computer packet losses cannot even be detected (beyond CRC mismatches).
While the Nspire's protocol doesn't have a retransmit scheme, each packet has a sequence number and a checksum.

[Egan Ford]

(01-05-2014 07:19 AM)debrouxl Wrote:  Could you provide the full terminal output, presumably through tee, of a failed upload and a successful upload ?

Attached.

[debrouxl]

Thanks
Diffing the output files shows that the two failures are identical, the two successes are identical, and the only difference between a failure and a success is that upon failure, the calculator replies nothing (no error code, for instance - the DBUS, DUSB and NSP protocols of the TI-Z80 series, TI-68k series and Nspire series have that), instead of a ready packet.

I won't be able to do much about it, only slightly reworking the code for checking whether the calculator sent back a "ready" packet (which could now be termed "ready / success", I guess), and return a special error code indicating transfer error if no such packet was returned to the computer before timeout. Currently, "ready" packets are swallowed.
If something in the USB HID stack (or the calculator side ??) cannot cope with more than several dozens of KBs of host -> device data, and silently discards or corrupts data, well...

[Cristian Arezzini]

I just tried the whole procedure again after installing a fresh Ubuntu system.
I followed the README in libhpcalc's page here.
The whole process went without errors or warnings, and at the first test (with sudo) I got the menu with the choices. I tried receiving a screenshot, but the process hung halfway.
Now when I try receiving a screenshot, these are the last lines I see:

Code:

hpcalcs DEBUG: Dumping IN packet with size 64
hpcalcs DEBUG:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
hpcalcs DEBUG:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
hpcalcs DEBUG:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
hpcalcs DEBUG:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
hpcalcs INFO: prime_recv_data: breaking because the expected size was reached (2)
hpcalcs INFO: prime_recv_data: shortening packet from 98595 to 98567
hpcalcs DEBUG: read_vtl_pkt: command matches returned data
hpcalcs INFO: calc_prime_r_recv_screen: embedded=DA63 computed=D15F
hpcalcs ERROR: calc_prime_r_recv_screen: CRC mismatch
hpcalcs WARN: calc_prime_r_recv_screen: unknown marker at beginning of image
hpcalcs ERROR: calc_prime_recv_screen: r_recv_screen failed
hpcalcs ERROR: hpcalcs_calc_recv_screen: recv_screen failed
hpcalcs_calc_recv_screen failed
hpcalcs INFO: Unhandled packet format

388 Unhandled packet format

or sometimes I get this:

Code:

Choose a format (for Prime, usually 8 to 11):8

8
Enter output filename: /home/cristian/prova

/home/cristian/prova
hpcalcs INFO: prime_send_data: q:0      r:2
hpcalcs DEBUG: Dumping OUT packet with size 3
hpcalcs DEBUG:     00 00 FC 
hpcables INFO: cable_prime_hid_send: wrote 0 bytes
hpcalcs INFO: prime_send: send succeeded
hpcalcs INFO: prime_send_data: send remaining succeeded
hpcables INFO: cable_prime_hid_recv: read 0bytes
hpcalcs INFO: prime_recv_data: breaking due to short packet (1)
hpcalcs INFO: prime_recv_data: shortening packet from 0 to 0
hpcalcs INFO: read_vtl_pkt: empty packet
hpcalcs INFO: calc_prime_r_recv_screen: packet is too short: 0bytes
hpcalcs ERROR: calc_prime_recv_screen: r_recv_screen failed
hpcalcs ERROR: hpcalcs_calc_recv_screen: recv_screen failed
hpcalcs_calc_recv_screen failed
hpcalcs INFO: Unhandled packet format

388 Unhandled packet format

or this:

Code:

Choose a format (for Prime, usually 8 to 11):8

8
Enter output filename: /home/cristian/prova

/home/cristian/prova
hpcalcs INFO: prime_send_data: q:0      r:2
hpcalcs DEBUG: Dumping OUT packet with size 3
hpcalcs DEBUG:     00 00 FC 
hpcables ERROR: cable_prime_hid_send: write failed (null)
hpcables WARN: hpcables_cable_send: send failed
hpcalcs ERROR: prime_send: send failed
hpcalcs INFO: prime_send_data: send remaining failed
hpcalcs ERROR: calc_prime_recv_screen: s_recv_screen failed
hpcalcs ERROR: hpcalcs_calc_recv_screen: recv_screen failed
hpcalcs_calc_recv_screen failed
hpcables INFO: Error writing to cable

259 Error writing to cable

and so on. Basically it's very unstable, I haven't been able to successfully transfer anything yet...

Sometimes the program doesn't even start and I get this:

Code:

cristian@cristian-notebook:~/lpg/hplp/hplp/libhpcalcs/tests$ ./test_hpcalcs
Entering program
hpfiles INFO: hpfiles library version 0.0.1 compiled on Feb  9 2014 10:12:41
hpfiles INFO: hpfiles_init: init succeeded
hpcables INFO: hpcables library version 0.0.1 compiled on Feb  9 2014 10:12:42
hpcables INFO: hpcables_init: init succeeded
hpcalcs INFO: hpcalcs library version 0.0.1 compiled on Feb  9 2014 10:12:42
hpcalcs INFO: hpcalcs_init: init succeeded
hpopers INFO: hpopers library version 0.0.1 compiled on Feb  9 2014 10:12:43
hpopers INFO: hpopers_init: init succeeded
Initialized libraries
hpcables INFO: hpcables_handle_new: handle allocation succeeded
hpcalcs INFO: hpcalcs_handle_new: calc handle allocation succeeded
hpcables ERROR: cable_prime_hid_open: cable open failed
hpcables ERROR: hpcables_cable_open: open failed
hpcalcs ERROR: hpcalcs_cable_attach: cable open failed
hpcalcs INFO: hpcalcs_handle_del: calc handle deletion succeeded
hpcables INFO: hpcables_handle_del: handle deletion succeeded
Exiting program
hpopers INFO: hpopers_exit: exit succeeded
hpcalcs INFO: hpcalcs_exit: exit succeeded
hpcables INFO: hpcables_exit: exit succeeded
hpfiles INFO: hpfiles_exit: exit succeeded
Goodbye world!

The last lines of dmesg are these:

Code:

[ 8575.872096] usb 1-5: new high-speed USB device number 18 using ehci-pci
[ 8576.005143] usb 1-5: New USB device found, idVendor=03f0, idProduct=0441
[ 8576.005162] usb 1-5: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 8576.005171] usb 1-5: Product: HP Prime Calculator
[ 8576.005179] usb 1-5: Manufacturer: Hewlett-Packard
[ 8586.004409] hid-generic 0003:03F0:0441.000B: timeout initializing reports
[ 8586.004885] hid-generic 0003:03F0:0441.000B: hiddev0,hidraw1: USB HID v1.01 Device [Hewlett-Packard HP Prime Calculator] on usb-0000:00:02.1-5/input0

[debrouxl]

Hi Cristian

About your traces:
1) "unknown marker at beginning of image" means that the

Code:

if (pkt->data[8] == (uint8_t)format && pkt->data[9] == 0xFF && pkt->data[10] == 0xFF && pkt->data[11] == 0xFF && pkt->data[12] == 0xFF)

boolean condition for the data sent by the calculator is false. In that case, I'd need the full dump to see the data sent by the calculator... but I doubt the calculator has sent anything remotely correct, seeing that the last packet is all zeros: that's no valid ending for a PNG file.

2) the calculator replied nothing in 2 seconds. That seemed to be a safe timeout value, as screenshots always took much less than this.

3) the calculator was available when the test_hpcalcs program was started (otherwise you wouldn't have reached this step), but it looks like the calculator somehow disappeared from under the program.

4) communication with the calculator couldn't even be started, as opening the device, in the underlying HID library, failed. libhpcalcs / test_hpcalcs fails gracefully in that event, there's nothing else it can do (besides crashing, but that's no good ).

Bugginess in libhpcalcs is possible, but other reports until now tended to indicate it can work more consistently... and the first and last errors, especially, are hard to attribute to libhpcalcs (if communication fails completely, or returns garbage data, it can't do much).
Are you sure your calculator is in a stable state ?

[Cristian Arezzini]

(02-09-2014 10:33 AM)debrouxl Wrote:  Are you sure your calculator is in a stable state ?

Not at all! :-) I actually forgot to mention that sometimes, when starting communication, the calculator would spontaneously reboot; other times it would hard-crash, forcing a paperclip reset. I suppose I'll try another cable... ;-)

[debrouxl]

The tests performed by critor on his Prime show that libhpcalcs still mostly works with the latest (revision 6030) firmware, so the foundation of the protocol clearly hasn't changed

* the ready check, get infos, send key and send chat operations still work;
* the get screenshot operation yields "unhandled packet format", I'll have to find out why based on the debugging logs produced by libhpcalcs.

I had started making some changes for skipping the metadata at the beginning of .hpprgm files, as well as for making the "receive backup" operation more reliable in the face of all those packet losses occurring with the previous firmware version.

[danielmewes]

I've fixed a couple of bugs in the protocol implementation.
You can find my fixes here https://github.com/danielmewes/hplp/
I've also submitted a pull request to debrouxl https://github.com/debrouxl/hplp/pull/2 , so the changes will hopefully find their way into the main repo eventually.

I can now backup my Prime (latest firmware 6975) on Linux and also send apps back to the calculator without any apparent corruption (tested with an app of 1.8 KB).
Thank you so much for this awesome software @debrouxl!

Screenshots work as well, though type 8 doesn't work for me. Type 9 gives the best results, even though the colors are somewhat off.

[danielmewes]

@Cristian Arezzini : Without the patches I got pretty much the same behavior that you were seeing (I'm also on Ubuntu by the way). You might want to give them a shot if you're still interested in getting this working.

[debrouxl]

Quote:I've fixed a couple of bugs in the protocol implementation.

Great

Quote:I've also submitted a pull request to debrouxl https://github.com/debrouxl/hplp/pull/2 , so the changes will hopefully find their way into the main repo eventually.

Sure. Bonus points for the detailed comments interspersed with the code changes, I think it's the first time I see such detail in a pull request.
Even though few people will want to use old firmware versions because they have far more bugs than versions 6940/6975 do, have you tested communication with a calculator running firmware versions 5447 and 6030 ?

Have modern firmware versions become sufficiently reliable (as in, extremely low packet loss rates) for a split receive backup operation to be lower priority than it used to be ? I could post the unfinished patch.

Quote:Thank you so much for this awesome software @debrouxl!

You're welcome
As indicated in the first post, I tried to make it reasonably correct (though your contributions shows it needs fixes ), portable and interoperable, so as to hopefully maximize its usefulness and usability.
It still lacks a portable Qt GUI, though...

Quote:Screenshots work as well, though type 8 doesn't work for me. Type 9 gives the best results, even though the colors are somewhat off.

Yeah, the image needs to be decompressed, then colors need to be transformed in a publicly documented way, then raw / PNG / whatever images can be produced. I never spent time actually doing the code...
I have bookmarked multiple detailed usage examples for libpng 1.2 over the Web:
http://zarb.org/~gc/html/libpng.html
http://blog.hammerian.net/2009/reading-p...om-memory/ (now seemingly broken)
http://www.piko3d.net/tutorials/libpng-t...m-streams/
libpng 1.2 has a verbose API, but it's still too early to switch to libpng 1.6, as it's not quite available to everybody (e.g. the many users of Debian and derivatives) out of the box.

[danielmewes]

Quote:have you tested communication with a calculator running firmware versions 5447 and 6030 ?

No, unfortunately I only have one prime and don't want to downgrade. It seems that the main thing that got fixed in the recent firmware are the package sequence numbers coming from the calculator.
To increase the chance that this still works with older firmware, we could remove the `break` in https://github.com/danielmewes/hplp/comm...cd4971L182 I suppose (and only print a warning).

Thanks for the links for post-processing the images! I might look into it some time later.

I couldn't detect any package loss during backup, but haven't checked very thoroughly. Generally communication seems quite stable though.


One more thing I just noticed (you probably already know this, but might be interesting for other readers): When transferring a .hppgrm file to the calculator, the Prime seems to expect a raw text file without any headers. A lot of programs available on the Internet for download (e.g. on hpcalc.org) have the binary headers in the hpprgm files, which must be stripped before transferring such a program to the Prime with your tool.

[debrouxl]

Quote:One more thing I just noticed (you probably already know this, but might be interesting for other readers): When transferring a .hppgrm file to the calculator, the Prime seems to expect a raw text file without any headers. A lot of programs available on the Internet for download (e.g. on hpcalc.org) have the binary headers in the hpprgm files, which must be stripped before transferring such a program to the Prime with your tool.

Heh. I also started making some code for skipping the leading binary metadata, more than 6 months ago, but never finished it either
The leading metadata is made of blocks prefixed with their size in bytes.

Attached is my local patchset on top of the master branch:
* patch in proper shape (in master2 branch) for adding tentative IDs and types for .hpappnote/.hpappprgm files;
* incorrect patch for communicating with the computer version of the Prime (mostly works, pretty useful for debugging under Wine; it leaves the connection in an improper state, requiring Prime computer software restarts), the computer version of the 39gII (works somewhat less properly), and real 39gII calculators (tested by Marcus Von Cube, doesn't work);
* unfinished patch which probably doesn't compile for splitting backup parsing, so that upon packet loss, a subset of the data can be salvaged by the library clients if so they wish, instead of returning nothing but an error to clients;
* unfinished patch which definitely doesn't compile, for skipping the leading binary metadata. The goal was to loop over the (size, data) blocks until the size bytes contain a value over the file's size, which would indicate that the text data was found. The UTF-16LE BOM skipping code needs to run after that.

There are at least two useful things that I didn't port over from libti* in the beginning, to keep matters simpler, but which are there in libti* for a good reason and should migrate to any well-behaved communication library (though with a different implementation) at some point:
* port IDs, which are used in libti* for computer-side emulators <-> real calculators communication, or e.g. TILP <-> emulator communication. I've never fully dug into how they work in libti*/tilp;
* progress bar information, though the libti* mechanism is over-complex.

[debrouxl]

I pulled Daniel's 5 commits into my master branch, then I pushed the tentative support for .hpappnote and .hpappprgm files made in 2013 but never confirmed or infirmed by anyone (AFAICT).
The master2 branch has been force updated with the more experimental commits for which I posted a tarball in my previous post.

Thanks again Daniel

[djwesttexas]

Has anyone successfully connected their HP-Prime to a mac using this package? If so, forgive me for being such a dunce, but how? I have downloaded the package, unzipped it and try to run the test_hpcalcs.exe file. This starts an app called "File Juicer", but there is nothing there.

[debrouxl]

Quote:Has anyone successfully connected their HP-Prime to a mac using this package?

Yes, multiple MacOS X users have used test_hpcalcs to successfully communicate with their calculators.

Quote:If so, forgive me for being such a dunce, but how?

They have compiled it by themselves.

Quote:I have downloaded the package, unzipped it and try to run the test_hpcalcs.exe file. This starts an app called "File Juicer", but there is nothing there.

That is a Windows executable file
Under Linux, I have a cross-compiler for Windows, but I don't have a cross-compiler for MacOS X; I don't have a Mac computer, they're way too expensive
I suppose I could bother e.g. Adriweb for building MacOS X binaries for libhpcalcs + test_hpcalcs, too.

[rennanrene]

Hi everyone,

I'm totaly beginner in this field . I installed and connected sucessfully my calculator with this package. Therefore when i send a file the content of file change to a serie of charactere diferent. Someone can help me with this issue, pls?

thank you.