libhpcalcs: portable (Windows / MacOS X / Linux) connectivity kit library

[debrouxl]

Since mid-October 2013, I've been working on reverse-engineering Prime communications (thanks to packet dumps sent by critor), and from that knowledge, implementing a lightweight backend for a FLOSS third-party Prime connectivity kit aimed at Windows, Linux and MacOS X (possibly FreeBSD and derivatives as well, the USB HID library I've chosen to use works on FreeBSD)
No dependency on a huge, bloated non-native framework.

libhpcalcs currently has support for the following operations, through a terminal-based interface ("test_hpcalcs"):
* sending files;
* receiving files;
* receiving backups;
* receiving screenshots;
* getting undecoded information about the calculator, including firmware version;
* setting the calculator's time;
* triggering some form of "ready" check, or something like that;
* sending keypresses.
(more is in the pipeline)

Development source code: https://github.com/debrouxl/hplp (master branch is stable, master2 contains changes for testing, which may be rewritten before integration to master)
Ready-made install script aimed at *nix: https://raw.github.com/debrouxl/hplp/mas...ll_hplp.sh. Unless your distro packages hidapi (Debian and derivatives do not), you'll have to compile it yourself (autotools-based program, so configure && make && sudo make install).
Latest Windows binaries + source tarball: http://tiplanet.org/beta/libhpcalcs-0.0.1-package.zip .
Program to be run after decompressing or compiling: test_hpcalcs(.exe).


The current state of the code base (written in C99 for interoperability and portability, BTW) is that:
* on the plus side, the aforementioned operations usually work, for my several beta-testers, on Windows, Linux and MacOS X. One of the operations is reported to choke on MacOS X, but it works on Linux and Windows;
* on the minus side, there's no post-processing of screenshots so that they have proper colors... and most of all, there's no GUI yet...

The architecture and API are derived from the time-proven libti* architecture. They can support the 39gII, which is closer to the Prime than, say, a TI-85 with BlackLink cable is to a Nspire with USB cable. Ditto for computer versions of the Prime.

The Prime might support more operations, but that's undetermined yet. For instance (those are operations available on some, or all, TI graphing calculators):
* quickly getting the list of files without triggering a lengthy full backup ("dirlist", an very important operation - IMO, if HP hasn't implemented it yet, they should do it !);
* remote file operations (delete, rename, maybe even calculator-side copy).


Tim Wessman needs to be thanked once more for publicly helping with the peculiarities of CRC computations and screenshots on the Prime
Thanks to the beta testers and users as well.

[Jonathan Cameron]

Ok, I have it downloaded and installed on Ubuntu (12.04). I had to install both hidapi first as you suggested and had to download all the extra packages that are mentioned on the README page. It was a bit tedious and I had to try it several times, but it eventually compiled. (It is a good thing I'm not afraid of the command line!)

When I run test_hpcalcs as regular user, connecting to the calculator fails (the "cable open failed" error). If I run as root, it works fine. I set up /etc/udev/rules.d so that the calculator usb port gets mounted with suitable permissions and it did not make any difference for regular users. Do you have any suggestions?

UPDATE: I was able to get a regular user to work by manually setting the permission of the raw hid node: /dev/hidraw2. My udev rule worked fine for the corresponding node: /dev/usb/hiddev1. Now I have to figure that out.

By the way, when I exit test_hpcalcs, I'm getting a seg-fault.

Thanks!

-Jonathan

[Jonathan Cameron]

I was able to get test_hpcalcs working for regular users on Ubuntu (12.04) by doing two things:

1. Add the user to the 'dialout' group

2. Create a new udev rule in /etc/udev/rules.d/80-hp-prime.rules:

Code:


# HP Prime Calculator
SUBSYSTEM=="usb",  ATTRS{idVendor}=="03f0", ATTRS{idProduct}=="0441", ACTION=="add", GROUP="dialout", MODE="0666"
KERNEL=="hidraw*", ATTRS{idVendor}=="03f0", ATTRS{idProduct}=="0441", ACTION=="add", GROUP="dialout", MODE="0666"

Notes: Using the group is probably not necessary. If you are in the group, you can use 664 for permissions, if you are not, 666 means everyone can read/write the port, including you. So if you are doing this on your own laptop, no group and MODE="0666" should be fine. Otherwise use the group approach shown here with MODE="0664".

-Jonathan

[Cristian Arezzini]

Hi Jonathan,
Would it be possible for you to write a step-by-step tutorial on how to install the package (and its dependencies) under Ubuntu? I'm not scared of the command line but I really don't have the free time to tinker with it...

Cristian

[Jonathan Cameron]

Cristian,

I've put togehter the instructions for installing hplp on Ubuntut 12.04. See the attachment. It also includes the udev mods I posted before. It is pretty much following the instructions in the README files except that I had to edit one Makefile.


  hplp-ubuntu-install.txt (Size: 3.31 KB / Downloads: 45)

-Jonathan

[Cristian Arezzini]

Hi Jonathan,
Thank you very much. I'm following your very clear tutorial, but I still have some issues (I'm currently running Ubuntu 13.04):
1) There is another prerequisite I needed to install, which is autopoint. Without that the procedure fails.
2) In the Makefile I had to edit, the required line was not at line 166 but at 188.
3) While executing "make check" on hplp, the test fails. I'm attaching below the output of the command.


Thanks again
Cristian


EDIT: the error is about a missing lib, libhidapi-hidraw.so.0 but running the locate command I get this:

Code:

root@cristian-laptop:/opt/hplp/libhpcalcs/tests/.libs# locate libhidapi-hidraw
/opt/hidapi/linux/libhidapi-hidraw.la
/opt/hidapi/linux/.libs/libhidapi-hidraw.a
/opt/hidapi/linux/.libs/libhidapi-hidraw.la
/opt/hidapi/linux/.libs/libhidapi-hidraw.lai
/opt/hidapi/linux/.libs/libhidapi-hidraw.so
/opt/hidapi/linux/.libs/libhidapi-hidraw.so.0
/opt/hidapi/linux/.libs/libhidapi-hidraw.so.0.0.0
/usr/local/lib/libhidapi-hidraw.a
/usr/local/lib/libhidapi-hidraw.la
/usr/local/lib/libhidapi-hidraw.so
/usr/local/lib/libhidapi-hidraw.so.0
/usr/local/lib/libhidapi-hidraw.so.0.0.0

so the lib appears to be there...

[Jonathan Cameron]

(12-15-2013 06:56 AM)Cristian Arezzini Wrote:  3) While executing "make check" on hplp, the test fails. I'm attaching below the output of the command.

I had forgotten that I had to install autopoint. I'll update the instructions.

Regarding the error:

Quote:/opt/hplp/libhpcalcs/tests/.libs/lt-torture_hpcalcs: error while loading shared libraries: libhidapi-hidraw.so.0: cannot open shared object file: No such file or directory

This looks like a linking problem. Either hidapi did not compile correctly (you'll have to double-check that), or the lib files went into some location that test_hpcalcs did not get informed about. Looking over the build log, I see that libhidap-hidraw.so* are supposed to be in /usr/local/lib. So your first step is to check to see if they are there. If not, you may need to rebuild/reinstall hidapi.

Send me a private message if you would like to work together on this.

-Jonathan

P.S. When you figure this out, I'd be glad to update the instructions for 13.04.

[debrouxl]

Thanks for making a detailed install procedure
With your permission, I'm simply going to adopt it for the README, with at least the following tweaks:
* in step 1, use the union of the package list mentioned in install_hplp.sh and your list;
* in step 3, use install_hplp.sh instead of manual cloning, configuring with wrong CFLAGS, and manual makefile edit

And thanks for the bug report and patch.

Cristian: the most common cause of this error is that LD_LIBRARY_PATH does not contain the folder where the shared object is. You need to add /usr/local/lib to LD_LIBRARY_PATH, or use --prefix=/usr when configuring hidapi.

[Jonathan Cameron]

(12-15-2013 07:22 AM)debrouxl Wrote:  Thanks for making a detailed install procedure
With your permission, I'm simply going to adopt it for the README, with at least the following tweaks:

You are welcome.

Feel free to modify and use the ubuntu install file however you like.

-Jonathan

---

When I got this working on Ubuntu, I was able to transfer a program file to my physical calculator (unfortunately, the contents are corrupted and truncated).

However, when I try to send an App program (MyApp.hpappprgm), the file structure is not recognized. Apparently the binary contents differ from what is expected (in the header part). Do you plan to look at handling App programs soon?

Thanks for your work on this.

-Jonathan

---

(12-15-2013 07:22 AM)debrouxl Wrote:  Thanks for making a detailed install procedure

You may want to download the ubuntu install instructions file again. I just updated it for some issues that Cristian encountered.

-Jonathan

[debrouxl]

Jonathan Cameron Wrote:When I got this working on Ubuntu, I was able to transfer a program file to my physical calculator (unfortunately, the contents are corrupted and truncated).

Is the corruption / truncation reproducable ?
Other users didn't report corruption, so there could have been a transfer error... or maybe the program was broken relatively recently
libhpcalcs makes no assumption or checking on the format of programs - it just sends bytes over the wire.

Quote:However, when I try to send an App program (MyApp.hpappprgm), the file structure is not recognized. Apparently the binary contents differ from what is expected (in the header part). Do you plan to look at handling App programs soon?

I do
In fact, on the aforementioned "master2" branch of the Git repository, for nearly a month, there has been a tentative patch adding support for .hpappnote and .hpappprgm. I don't have a Prime myself, so I'm relying on beta-testers, to whom I posted about the change... but nobody has reported success or failure yet.
It would be great if you could be the first such beta-tester

[Jonathan Cameron]

(12-15-2013 07:54 AM)debrouxl Wrote:  In fact, on the aforementioned "master2" ...
It would be great if you could be the first such beta-tester

I'll give it a try, but it may be a few days.

-Jonathan

[Cristian Arezzini]

I tried using the automatic install script, and the installation appears to have worked. (I had also included Jonathan's diff patch).
But upon running, the program still complains about the missing lib:

Code:

root@cristian-laptop:/opt/hplp/libhpcalcs/tests# ./test_hpcalcs 
/opt/hplp/libhpcalcs/tests/.libs/lt-test_hpcalcs: error while loading shared libraries: libhidapi-hidraw.so.0: cannot open shared object file: No such file or directory

Where exactly is that LD_LIBRARY_PATH field I should update? I'm sorry but I'm really new at compiling stuff...

[debrouxl]

LD_LIBRARY_PATH contains a list of folders separated by ":", and is used by the dynamic linker (part of the program loading sequence) to find shared objects.
Alternatively, /etc/ld.so.conf can be modified, or files can be added to the /etc/ld.so.conf.d folder. Under my Debian sid install, there's a file in /etc/ld.so.conf.d which points to /usr/local/lib.

[Cristian Arezzini]

Thanks, but what I don't understand is: where do I find that LD_LIBRARY_PATH? I suppose it's an entry in a file, but which file?

---

OK, I found out that LD_LIBRARY_PATH is a system variable, so I added the library folder with the command

Code:

export LD_LIBRARY_PATH=/usr/local/lib:${LD_LIBRARY_PATH}

but after recompiling I still get the same error.

[Cristian Arezzini]

Sorry for flooding the thread, but I found the problem. I had set LD_LIBRARY_PATH for my user, not for root. After setting it for root, the program runs (I still haven't tried any actual transfer, but I do get the commands menu).
So, right before running install_hplp.sh, I ran the command

Code:

export LD_LIBRARY_PATH=/usr/local/lib:${LD_LIBRARY_PATH}

and after compiling it worked.

[Cristian Arezzini]

I tried some transfers... they don't work. If I send an hpprgm to the calculator, I see some activity on the terminal, and then I see the program's name in the calc's programs list... but there's nothing inside except a single "&" character.
I also tried the screen capture, but that too doesn't work...

[debrouxl]

Hmm. Two people reporting problems is getting worrying...
But nothing relevant was changed in the code base since mid-November, where it worked, according to multiple users ( later pages of http://tiplanet.org/forum/viewtopic.php?t=13337 , in French)

I can't test for myself, I have no Prime...

[Cristian Arezzini]

Do you want me to post the output from an attempted transfer?

[debrouxl]

Sure
The library trace is the most direct way for me to try and understand what's going on.
After that, if we still don't understand, USB dumps, preferably through Wireshark, could provide more data ( http://wiki.wireshark.org/CaptureSetup/USB ).

I have processed Jonathan's patch, I am integrating his README changes and will make a commit later.

[Cristian Arezzini]

I'm attaching the output of a program transfer. I cut away most of the middle part since it was repetitive and too big to post.
I'm not a programmer but if you want me to do some experiments just let me know!

Cristian