libhpcalcs: portable (Windows / MacOS X / Linux) connectivity kit library

[debrouxl]

Looks like the entire program (or so) gets transferred, but nothing clearly wrong occurred.

As for screenshots, I forgot to mention that only formats 8 to 11 will yield a non-unusable (*) result. I've just edited the message in test_hpcalcs.c, the change will be in the next commit.

*: inverted colors in 4 bpp mode, and a grayscale image in 16 bpp mode. I know what post-processing needs to be done, but I haven't implemented it yet.

EDIT: I have pushed a commit with multiple fixes, the improved README, and an indication of the (most) valid screenshot formats

[debrouxl]

Jonathan, Cristian: there has been some testing transferring DJ's Tunnel game ( http://tiplanet.org/forum/archives_voir.php?id=19577 ) to the Prime through libhpcalcs, and it is indeed corrupted...

However, the problem goes away after hex-editing out the binary header, before the main EXPORT command (starting at offset 0x46, for that particular program). This means I'll have to detect and strip out this header (just like I'm already stripping the UTF-16LE BOM created by several editors, because the calculator's program interpreter chokes on it).
For now, the workaround is to strip that header manually with a hex editor... well, better than nothing

I've pushed two commits tonight, neither changes functionality in a visible manner.

[Jonathan Cameron]

(12-17-2013 09:02 PM)debrouxl Wrote:  I've pushed two commits tonight, neither changes functionality in a visible manner.

Sounds like good progress. I was doing some hex dumps on the files and noticed that my editor puts in the BOM (FFFE). Not surprised it is due to an issue like that. I look forward to testing your changes.

Which branch are you doing these changes on?

-Jonathan

[debrouxl]

The non-functional changes have been committed to the master branch.
I have rebased master2, where the untested commit for adding tentative IDs for .hpappnote and .hpappprgm is, onto master, so you can use that if you want to test .hpappprgm

I hope that .hpappprgm won't have the same aforementioned problem as .hpprgm. That said, for the narrow purposes of confirming the internal IDs for the .hpappnote and .hpappprgm files (probably 7 and 8), the test would be successful upon proper creation of a file of the corresponding type, no matter what the file's actual contents are, on the computer and calculator side.

[debrouxl]

Not that it's a highly important feature, but I have added support for the "send chat" operation.
The calculator can send chats as well, so there shall be a "recv chat" operation, left for later work.

[Tugdual]

What would be the benefit (apart from culture) of this tool compared with the standard Connection Kit?

[Cristian Arezzini]

The fact that it's multi-platform. For me for example, it will be the only way to connect to my prime. Right now, since it doesn't work well yet, I just can't transfer programs to my calculator..

[debrouxl]

Indeed, the main practical upside of libhpcalcs are its portability and interoperability.
* portability: it's the only way users of MacOS X, Linux and probably FreeBSD + some of its derivatives can communicate with their calculators without relying on Windows. Wine doesn't provide a USB stack.
* interoperability: pretty much every programming language can interoperate with C. Front-ends to the libhpcalcs backend can be written into pretty much any language, either directly (C, C++) or through wrappers which can be auto-generated (SWIG, FFI) for all of the most used programming languages (Python, Lua, Perl, etc.)
Additionally, libhpcalcs is free software (both in beer and in freedom), but it doesn't matter in day to day usage.

The main practical downside of libhpcalcs, for now, is that its usage remains non-graphical, as nobody bothers writing a GUI for libhpcalcs. To me, functionality, portability and interoperability have higher priority than UI work. I don't like doing things in the wrong order


Cristian: have you suppressed the leading metadata from your programs, as I mentioned above ? Seems to work around the problem, until I implement stripping of said metadata.
Transferring .hpprgm files worked for my beta-tester, and other persons later - otherwise, the feature would never have been released

[Cristian Arezzini]

(12-23-2013 06:48 AM)debrouxl Wrote:  Cristian: have you suppressed the leading metadata from your programs, as I mentioned above ? Seems to work around the problem, until I implement stripping of said metadata.
Transferring .hpprgm files worked for my beta-tester, and other persons later - otherwise, the feature would never have been released

It seems that I somehow missed your suggestion about the metadata! Sorry. I'll test as soon as possible and report here...

[LarsF]

I have manage to compile the package on my mac after som problems with libpng12. Receive files seems to work, send a file crash test_hpcalcs, but after reading the previous messages, that seems to be a bug. But i wonder, is it possible to list the content on the calculator, so get a list of programs/apps and so on?

btw, nice initiative to make something like this for us that don't run windows.

[debrouxl]

Quote:I have manage to compile the package on my mac after some problems with libpng12.

Good.

Quote:Receive files seems to work, send a file crash test_hpcalcs, but after reading the previous messages, that seems to be a bug.

Nope, the leading metadata is another issue, which does not (AFAICT) trigger crashes. It's not the first time a crash is reported on MacOS X, but that was another issue, AFAIK...
Could you launch test_hpcalcs through GDB and post a backtrace of the crash, i.e. `gdb test_hpcalcs`, then "run" when the prompt appears, then "bt" when the debugger receives control again after the program crashed ?

Quote:But i wonder, is it possible to list the content on the calculator, so get a list of programs/apps and so on?

Only indirectly, through receiving a full backup of the calculator, which was one of the first operations implemented by libhpcalcs. I don't know of any quick way to obtain the list of files, types and sizes (an operation known as "dirlist" on TI-Z80, TI-68k and TI-Nspire calculators). I assume that if such an operation existed, the HP CK would use it... and I really think that HP would do a service to its users by implementing such an operation (at least, for non-real/complex variables, those are very small anyway)
The full backup operation is slow, due to the transfer being rate-limited (~3-12 KB/s, as shown by the USB dumps: ~5 to ~20 ms between each 64-byte packet) on the calculator side, in order not to overwhelm Windows' poor little USB HID stack...

Quote:btw, nice initiative to make something like this for us that don't run windows.

Thanks.

[LarsF]

After som struggle to get gdb to run on mac, (its not part of command-line tools anymore...) I get this output of bt

(gdb) bt
#0 0x00007fff8721e3b6 in ?? ()
#1 0x00007fff5fbffa50 in ?? ()
#2 0x000000010000983e in ?? ()
#3 0x00007fff5fbffa6f in ?? ()
#4 0x00000001004000d0 in ?? ()
#5 0x0000000000000000 in ?? ()

I hope that mean something to you :-)

[debrouxl]

Raw addresses, without symbolic debugging information, are basically undecipherable
Unless you didn't use install_hplp.sh, I don't understand why debug info is missing: install_hplp.sh explicitly passes -g3 among the CFLAGS to configure, triggering generation of a fairly high amount of debug info, and -s is intentionally not passed (it would cause debug info stripping) or added by configure.

[LarsF]

I'm sorry, i could not use the install script because of the libpng struggle, I will try to remake the package so I get debug info into the executable. I'll be back :-)

[LarsF]

I could use lldb to debug it and get this info

Enter input filename: MandA.hpprgm
Input file has size 1254 (4e6)
Process 51083 stopped
* thread #1: tid = 0x2e2aa, 0x00007fff8721e3b6 libsystem_c.dylib`strrchr + 10, queue = 'com.apple.main-thread, stop reason = EXC_BAD_ACCESS (code=1, address=0x2001600)
frame #0: 0x00007fff8721e3b6 libsystem_c.dylib`strrchr + 10
libsystem_c.dylib`strrchr + 10:
-> 0x7fff8721e3b6: movsbl (%rdi), %edx
0x7fff8721e3b9: cmpl %ecx, %edx
0x7fff8721e3bb: cmoveq %rdi, %rax
0x7fff8721e3bf: incq %rdi
(lldb) bt
* thread #1: tid = 0x2e2aa, 0x00007fff8721e3b6 libsystem_c.dylib`strrchr + 10, queue = 'com.apple.main-thread, stop reason = EXC_BAD_ACCESS (code=1, address=0x2001600)
frame #0: 0x00007fff8721e3b6 libsystem_c.dylib`strrchr + 10
frame #1: 0x00000001000099df libhpcalcs.0.dylib`prime_parsefilename(filepath=<unavailable>, out_type=0x00007fff5fbffa8f, out_calcfilename=0x00007fff5fbffa80) + 47 at typesprime.c:132
frame #2: 0x0000000100000e00 test_hpcalcs`send_file(handle=0x000000010010faf0) + 235 at test_hpcalcs.c:263
frame #3: 0x0000000100000a36 test_hpcalcs`main(argc=<unavailable>, argv=<unavailable>) + 375 at test_hpcalcs.c:583
frame #4: 0x00007fff8e43e5fd libdyld.dylib`start + 1
frame #5: 0x00007fff8e43e5fd libdyld.dylib`start + 1

[debrouxl]

Thanks
An access fault in strrchr means that its argument points to nowhere sane... which, in turn, means that basename returned something unexpected to me, possibly NULL.
Under Linux, `man basename` shows no relevant way for dirname() and basename() to return NULL, so I didn't bother adding a check. What does `man basename` say on MacOS X ?

Also, for testing purposes, try replacing

Code:

char * file = basename(filepath);
char * extension = strrchr(file, '.');

by something along the lines of

Code:

char * duplicated = strdup(filepath);
char * file = basename(duplicated);
char * extension = strrchr(file, '.');
free(duplicated);

(I'm going to do something pretty much that, with more error checking, for portability purposes)

[Egan Ford]

I just started testing this on OS/X. I used get clone today to download, and autoreconf/configure/make instead of the .sh scripts. I also had to build hidapi first. libpng12 is old, using 1.5.x. I had to change configure.ac to use libpng instead of libpng12. I also had to use -std=c99.

So far I have been able to get screen shots. Type 9 looks the best, but the colors do not match. Dunno if this is due to libpng version or not. PNG output is 4 bit color. That could be it as well.

I can download (from the calc), but not upload programs. Programs download in UTF-16LE encoding. After running iconv -f UTF-16LE -t UTF-8 foo.hpprgm >foo.txt I was able to edit files with vi (assuming LANG=en_US.UTF-8). After converting back to UTF-16LE and trying to upload I get:

Code:


Enter input filename: bar.hpprgm
Input file has size 3054 (bee)
Segmentation fault: 11

If I just try to transfer back what I downloaded, I still get the same error.

I've had a few stability issues as well. Hard to tell if it's the Prime or the code. After disconnecting and reconnecting the cable all is fine. Once the Prime reset itself.

Thanks for this tool. I imagine in a month or too I'll be able to develop only on OS/X.

[Egan Ford]

(12-26-2013 12:20 AM)Egan Ford Wrote:  I can download (from the calc), but not upload programs. Programs download in UTF-16LE encoding. After running iconv -f UTF-16LE -t UTF-8 foo.hpprgm >foo.txt I was able to edit files with vi (assuming LANG=en_US.UTF-8). After converting back to UTF-16LE and trying to upload I get:

Code:


Enter input filename: bar.hpprgm
Input file has size 3054 (bee)
Segmentation fault: 11

If I just try to transfer back what I downloaded, I still get the same error.

Easy fix, patch for OS/X, put:

Code:


#ifdef __APPLE__
#include <libgen.h>
#endif

in typesprime.c, this will get basename working. I didn't catch the warning on compile.

Now I can send and recv files on OS/X. Thanks again.

[debrouxl]

Hi Egan,

Quote:I just started testing this on OS/X. I used get clone today to download, and autoreconf/configure/make instead of the .sh scripts.

Which caused you to add -std=c99 yourself
Since pretty much everyone using autoreconf/configure/make directly hits the problem, I'm adding AC_PROG_CC_C99 to configure.ac, and AC_MSG_ERROR(...) if ac_cv_prog_cc_c99 is "no", as described in the autoconf manual. Will commit after more testing, if successful.

Quote:I also had to build hidapi first. libpng12 is old, using 1.5.x. I had to change configure.ac to use libpng instead of libpng12.

libpng being mostly backwards-compatible, I guess that I'll do the same.

Quote:So far I have been able to get screen shots. Type 9 looks the best, but the colors do not match. Dunno if this is due to libpng version or not. PNG output is 4 bit color. That could be it as well.

It's not due to the libpng version, it's due to the way the Prime encodes screenshots
It's publicly described, it's been in the TODO list since the beginning, but I haven't implemented the necessary post-processing yet. libpng has a versatile, but mildly complex, API, and for now, I'm mainly onto something else.

Thanks for the information on fixing send_file. Locally, I've now combined both #include <libgen.h>, which is the standard way to get basename even on Linux, and the code I posted in my previous post (with extra refactoring). Will commit after more testing.

[LarsF]

After I added the lines
#ifdef __APPLE__
#include <libgen.h>
#endif
then the sending start working too. I also miss the warning on basename.
Thanks