Trying to improve x49gp

[3298]

(08-29-2018 02:17 AM)Claudio L. Wrote:  I was getting and error with the FORTIFY_SOURCE being redefined, so I removed from the Makefile, I don't know if it can affect other platforms (I only tested on Linux).

Well, that's a bummer. I added that because it somehow suppressed the stringop-truncation warning on my system (Manjaro 64-bit, fully updated). Maybe that definition caused my GCC to switch to another version of strncpy which isn't marked for that warning, but if adding it explicitly breaks on your setup, I need to find a better fix.
A closer look at the only instance of strncpy in the entire codebase reveals that the limit it's given is always shorter than the source string, so the destination string will never get a terminating null character. In other words, it's used as a memcpy and can be switched to actually use memcpy instead without side effects other than a bogus warning going away. Let's do it.

(08-29-2018 02:17 AM)Claudio L. Wrote:  I also tested it as a debugger, and it worked much better than previously. Now it's fast and responsive, I can inspect any kind of variables, etc.
The only issue is that stepping over one line sometimes jumps many lines, as in goes overboard. For example I single-stepped one line within a loop, and by the time it stopped it had finished the loop and advanced 5 or 6 lines past the end of the loop. But at least now there's no errors when setting and removing breakpoints, and it actually stops at breakpoints (seems trivial but not always worked before).

Ouch, that's not supposed to happen. I somehow doubt it's the interrupt fix skipping too far, because the fix works by preventing interrupts from starting in the first place (instead of letting them run and stopping afterwards), but the only other place for an actual bug I can think of is the gdb client, which is obviously not part of x49gp.
The knowledge about where code lines start and end is only on the client side, which automatically translates a line step into a stream of alternating instruction-step commands and register queries (to check the instruction pointer). If there are too many of these, x49gp's gdbstub cannot even tell there's something wrong
There is another possibility, which is the compiler's optimization making a mess of your to-be-debugged code. That's happened to me more than a few times, but I know of only two ways to deal with it: turn it off by compiling your program / firmware with -O0, or accept it and try to make sense of the machine code it spat out.

Anyway, I got your requested features (automatic firmware installation via GUI, forced automatic firmware installation via command-line option, forced reboot) done. It's split into three patches (firmware installation code with GUI-based selection; all command-line additions; manual update). The fourth one is the strncpy->memcpy workaround above.

My commit messages:

Code:

Improve initial flash creation
Instead of just loading the bootcode and letting it handle the rest (relying on the user to provide a firmware through the emulated SD card),
ask the user for the firmware and load that into the flash directly.
Also, the stock firmware needs some markers to be able to use the user flash banks.
It seems to be unable to add these if they are missing, so if a firmware that does not include blank user banks is selected,
the user banks would have been unusable.
Adding the marks in x49gp is therefore desirable, and custom firmware needs to be robust enough
to handle these anyway if it wants to survive in the wild.
If the flash file was present but too short, there is most likely nothing worth preserving in it,
so stop trying and fill the entire flash with 0xff instead of just the section after the present file's contents.
Lastly, fix a double free on a rare (fatal) error and a use-after-free on another which appear to have snuck into the flash creation code in an earlier commit.

Code:

Add command-line options to rebuild the flash or reboot
The firmware can be specified on the command line.
This might be useful for debugging custom firmware by making the debugging process more automatable.

Code:

Update the manual to reflect recent changes

Code:

Fix a warning about strncpy that caused compilation failures via -Werror
The previous workaround through -D_FORTIFY_SOURCE=1 caused redefinition errors on a user's system,
and it working at all was probably an oversight in the development system's compiler or standard library headers.
The code that actually caused it can be safely switched to memcpy in place of strncpy, which looks like a better fix.

[Claudio L.]

Patch 32 didn't apply. I guess nobody has the manual x49gp.man.in but you.

[Claudio L.]

(08-29-2018 11:12 AM)3298 Wrote:  Ouch, that's not supposed to happen. I somehow doubt it's the interrupt fix skipping too far, because the fix works by preventing interrupts from starting in the first place (instead of letting them run and stopping afterwards), but the only other place for an actual bug I can think of is the gdb client, which is obviously not part of x49gp.
The knowledge about where code lines start and end is only on the client side, which automatically translates a line step into a stream of alternating instruction-step commands and register queries (to check the instruction pointer). If there are too many of these, x49gp's gdbstub cannot even tell there's something wrong
There is another possibility, which is the compiler's optimization making a mess of your to-be-debugged code. That's happened to me more than a few times, but I know of only two ways to deal with it: turn it off by compiling your program / firmware with -O0, or accept it and try to make sense of the machine code it spat out.

You are right! The project file on newRPL was removing the 'debug' configuration, so it was always -O2. I fixed the project files now it works perfectly. Debugging in ARM seems to be a lot jumpier than debugging directly on x86 for some reason. I guess the compiler still messes up the code a lot more, even with -Og selected, but it's not the insanity that was before.
Now it's a very nice debugger!

Anyway, I got your requested features (automatic firmware installation via GUI, forced automatic firmware installation via command-line option, forced reboot) done. It's split into three patches (firmware installation code with GUI-based selection; all command-line additions; manual update). The fourth one is the strncpy->memcpy workaround above.

My commit messages:

Code:

Improve initial flash creation
Instead of just loading the bootcode and letting it handle the rest (relying on the user to provide a firmware through the emulated SD card),
ask the user for the firmware and load that into the flash directly.
Also, the stock firmware needs some markers to be able to use the user flash banks.
It seems to be unable to add these if they are missing, so if a firmware that does not include blank user banks is selected,
the user banks would have been unusable.
Adding the marks in x49gp is therefore desirable, and custom firmware needs to be robust enough
to handle these anyway if it wants to survive in the wild.
If the flash file was present but too short, there is most likely nothing worth preserving in it,
so stop trying and fill the entire flash with 0xff instead of just the section after the present file's contents.
Lastly, fix a double free on a rare (fatal) error and a use-after-free on another which appear to have snuck into the flash creation code in an earlier commit.

Code:

Add command-line options to rebuild the flash or reboot
The firmware can be specified on the command line.
This might be useful for debugging custom firmware by making the debugging process more automatable.

Code:

Update the manual to reflect recent changes

Code:

Fix a warning about strncpy that caused compilation failures via -Werror
The previous workaround through -D_FORTIFY_SOURCE=1 caused redefinition errors on a user's system,
and it working at all was probably an oversight in the development system's compiler or standard library headers.
The code that actually caused it can be safely switched to memcpy in place of strncpy, which looks like a better fix.

[/quote]

[3298]

(08-29-2018 10:02 PM)Claudio L. Wrote:  Patch 32 didn't apply. I guess nobody has the manual x49gp.man.in but you.

Er, what? Patch 23 added it, if I counted right. Last one of the big package.
It's indeed missing in your repository, but other parts of the same patch got in. Apparently you didn't 'git add' it (and I forgot to mention that in the post with the patches). As a matter of fact, x49gp.desktop.in (from patch 19) is missing too, and the README and README.QEMU files I renamed to TODO and TODO.QEMU in patch 23 went poof as well. Sorry about not catching that earlier.
Edit: fixed the old post to warn any latecomers who want to recreate the entire Git history in their repositories. Better late than never.

[brickviking]

(08-29-2018 10:21 PM)3298 Wrote:  

(08-29-2018 10:02 PM)Claudio L. Wrote:  Patch 32 didn't apply. I guess nobody has the manual x49gp.man.in but you.

Er, what? Patch 23 added it, if I counted right. Last one of the big package.
It's indeed missing in your repository, but other parts of the same patch got in. Apparently you didn't 'git add' it (and I forgot to mention that in the post with the patches). As a matter of fact, x49gp.desktop.in (from patch 19) is missing too, and the README and README.QEMU files I renamed to TODO and TODO.QEMU in patch 23 went poof as well. Sorry about not catching that earlier.
Edit: fixed the old post to warn any latecomers who want to recreate the entire Git history in their repositories. Better late than never.

Just as well I refreshed this thread then, I had read the post before you "fixed" it, just before you posted this message I'm replying to.

I will be rather grateful for a slightly modernised x49gp, as a couple of the warts from other versions are causing me to have three directories - source, hp50g and hp49g+ just so I can get the bits I want. Let me know when the patch-fest eases up so I can get the codebase on my machine stabilised. Last time I did that was from the Sourceforge files, I think. Has anyone considered upgrading that codebase, or are we just going with the various git repositories?

(Post 271)

[3298]

(08-29-2018 10:46 PM)brickviking Wrote:  Just as well I refreshed this thread then, I had read the post before you "fixed" it, just before you posted this message I'm replying to.

I will be rather grateful for a slightly modernised x49gp, as a couple of the warts from other versions are causing me to have three directories - source, hp50g and hp49g+ just so I can get the bits I want. Let me know when the patch-fest eases up so I can get the codebase on my machine stabilised. Last time I did that was from the Sourceforge files, I think. Has anyone considered upgrading that codebase, or are we just going with the various git repositories?

Sourceforge is abandoned, as far as I know. All the good stuff is happening on GitHub, as well as right here in this thread, due to me not agreeing with GitHub's Terms of Service and Privacy Policy.
As for when would be a good time to upgrade: I'm pretty sure I didn't break anything important, so just go ahead. Cloning Claudio's repository is probably the easiest way out, even though his history doesn't fully match mine (he inserted a few quick hacks whenever I broke compilation for him, and he squashed the big pile of patches into one). The only actual files that are different from my local repository are the four missing files I just mentioned, and a file I deleted that still remains on Claudio's side (qemu/qemu-0.9.0.tar.gz). I just verified that by pulling a diff between a clone of his repository and mine, so this time we can be sure.

If you're on Linux, you should be good to go. I don't have a Mac, so it's completely untested for that, but it should just work anyway. (I have no clue how installing software works over there, but the Makefiles of other projects I've peeked into don't do any special handling of OSX, so I didn't do any either.) Just clone Claudio's repository, preferably extract the attached archive to restore the missing files (they aren't vital, but at least for installation they are good to have), and see if it compiles.
If it doesn't, yell at me. Otherwise, enjoy your new emulator, as you passed the biggest hurdle - compilation is the one thing that does break somewhat regularly.

Since you apparently have multiple instances, you can still use multiple config files to keep them separate. The only thing you may have to fix is the entry that gives one of them the 49g+ skin. I also recommend setting their name properties which gets used as the window's title to something recognizable, that may help tell them apart. Yes, the window is undecorated, but there are other places the window title is used, most prominently the window bar.
If you start the emulator without specifying any config file on the command-line, it'll use $HOME/.x49gp/config by default. You can just put the files of your main instance there.

---

Claudio, the archive may be helpful for you as well. Saves you the trouble of reconstructing the missing files. Patch 32 is integrated already.

[Claudio L.]

(08-29-2018 11:53 PM)3298 Wrote:  Claudio, the archive may be helpful for you as well. Saves you the trouble of reconstructing the missing files. Patch 32 is integrated already.

Done. Added all missing files and committed to my github.

To brickviking: go ahead and clone my repo, it's the one ahead of the pack for the time being. I will submit yet another pull request to Eddie but he hasn't pulled my previous ones yet, he may not like what we did with it. It's much more usable now.
I'll also see if Egan wants to update the sourceforge repos.

[brickviking]

That's great, it's looking good. My hp50g fires up fine after I go prod the 50g firmware (2.15) into it.
The following things left for me:

• I'd like the ability to fire up a hp49g+; do I simply swap out the png, or do I have to tweak the config to use the 49g+ firmware? (Is there any practical binary difference?)
• Additionally, how do I select this at runtime, i.e. x49gp -49|-50?
• Related to the first item, how would I "reset" the firmware during running, i.e. it boots up as a 50g, but I want to have a 49g+ without shutting down, erasing ~/.x49gp/* and reloading?
  

Sorry if I'm missing something simple here. About the only way I can see to do what I would like is x49gp -config 49+-config, and I still have to source a 49g+ firmware, I'm not sure I actually have one.

(Post 272)

[Claudio L.]

(08-31-2018 12:10 AM)brickviking Wrote:  That's great, it's looking good. My hp50g fires up fine after I go prod the 50g firmware (2.15) into it.
The following things left for me:

• I'd like the ability to fire up a hp49g+; do I simply swap out the png, or do I have to tweak the config to use the 49g+ firmware? (Is there any practical binary difference?)
• Additionally, how do I select this at runtime, i.e. x49gp -49|-50?
• Related to the first item, how would I "reset" the firmware during running, i.e. it boots up as a 50g, but I want to have a 49g+ without shutting down, erasing ~/.x49gp/* and reloading?
  

Sorry if I'm missing something simple here. About the only way I can see to do what I would like is x49gp -config 49+-config, and I still have to source a 49g+ firmware, I'm not sure I actually have one.

(Post 272)

Basically, the only thing you'd need to change is the background image, everything else is identical in both machines, as far as the emulator is concerned.
There's no difference in the firmware between 49g+ and 50g, except the text that appears when you use the VERSION command. The differences were mainly in the hardware (USB power, 4 batteries, tougher keys, etc), which is obviously meaningless in the emulator.

[3298]

(08-31-2018 12:10 AM)brickviking Wrote:  I'd like the ability to fire up a hp49g+; do I simply swap out the png, or do I have to tweak the config to use the 49g+ firmware? (Is there any practical binary difference?)

You go into the config and replace the line reading "type=hp50g" by "type=hp49g+". Editing the line reading "name=HP 50g" is optional, but may avoid confusion (if you delete that line entirely, it will be regenerated as "name=HP 50g" or "name=HP 49g+" depending on the value of the "type=" line, so you can also just remove it). x49gp will then use the other .png file for its appearance.

(08-31-2018 12:10 AM)brickviking Wrote:  Additionally, how do I select this at runtime, i.e. x49gp -49|-50?

Not possible. The type is part of the calculator state, so it belongs into the config.

(08-31-2018 12:10 AM)brickviking Wrote:  Related to the first item, how would I "reset" the firmware during running, i.e. it boots up as a 50g, but I want to have a 49g+ without shutting down, erasing ~/.x49gp/* and reloading?

Not possible either. I don't really see the point of it, though.

(08-31-2018 12:10 AM)brickviking Wrote:  and I still have to source a 49g+ firmware, I'm not sure I actually have one.

The 49g+ uses the same firmware as the 50g, so you definitely have one.

(08-31-2018 12:47 AM)Claudio L. Wrote:  Basically, the only thing you'd need to change is the background image, everything else is identical in both machines, as far as the emulator is concerned.
There's no difference in the firmware between 49g+ and 50g, except the text that appears when you use the VERSION command. The differences were mainly in the hardware (USB power, 4 batteries, tougher keys, etc), which is obviously meaningless in the emulator.

In addition to that, I believe the 49g+ had one less communication port (was it IR or serial? Don't remember). That's irrelevant too, because x49gp does not implement any communication to the outside world other than the SD card.

When creating a flash image from the firmware, x49gp also uses one of two different bootloader versions, depending on the calculator type. I assume they were dumped from real calculators of their respective types, and it's appropriate that the 49g+ gets an older version than the 50g (3.15.04 vs 4.01.03.03). I'm pretty sure they are irrelevant during operation, as the bootcode has nothing to say after it successfully starts the firmware.
Oh, the serial numbers (stored at the end of the bootloader) are also different: DE00000001 for the 49g+. DEA0000001 for the 50g.

That's it for the differences between 49g+ and 50g in x49gp: appearance and bootloader, including serial number. There are absolutely no other special cases in the code. If you change the type in the config without deleting the flash, you can even get a calculator with the 'wrong' bootloader and serial number, but it'll work just fine.

-----
Edit:
I thought of two more things I could improve in the context of flash creation.

Code:

When loading a firmware during flash initialization, check for the KINPOUPDATEIMAGE signature, and fail if it's missing

Code:

When selecting a firmware interactively, show errors in a message window and retry
The user may not even have a terminal open to catch the error message, so the previous behavior was basically a silent failure to them.

Makes sense, doesn't it?

[Claudio L.]

(08-31-2018 08:15 AM)3298 Wrote:  I thought of two more things I could improve in the context of flash creation.

Code:

When loading a firmware during flash initialization, check for the KINPOUPDATEIMAGE signature, and fail if it's missing

Code:

When selecting a firmware interactively, show errors in a message window and retry
The user may not even have a terminal open to catch the error message, so the previous behavior was basically a silent failure to them.

Makes sense, doesn't it?

You are on a roll!
I thought of another cool thing you can do to this emulator:
hp39gs and hp40gs emulation!
They have more or less the same hardware, I think the key areas are:
* Flash and RAM size: I believe the firmware doesn't detect size of flash or ram, it's hardcoded so it should work without modifications.
* Bootloader: I don't know if there's a bootloader binary for these machines somewhere. The KINPOUPDATEIMAGE signature is different, so it needs the different bootloader to accept the firmware (although we could modify the signature when we load the firmware, I think the 50g bootloader should flash it without any problems).
Screen: The screen of course is a few pixels smaller (64 vs 80), so that will require tweaking the emulation code a little.
SD card emulation: Can be left enabled, it won't do any harm I think (so nochanges needed to x49gp).
Keyboard: We need a new PNG and remapping of all the key functions labels, this is the most tedious work I think. But the keyboard emulation per se doesn't need to be changed at all.

[pier4r]

little note. Sourceforge did bad stuff years ago but since 2015/2016 improved again.

Github is github.
Then one has bitbucket, assembla, etc...

[brickviking]

(08-31-2018 03:22 PM)3298 Wrote:  In addition to that, I believe the 49g+ had one less communication port (was it IR or serial? Don't remember). That's irrelevant too, because x49gp does not implement any communication to the outside world other than the SD card.

I thought that x49gp used the "usb" interface on the machine to connect to its 50g instance? Or can't I connect to it with ckermit after all? Maybe that's what's been crashing my machine every time I try it.

(08-31-2018 07:43 PM)pier4r Wrote:  little note. Sourceforge did bad stuff years ago but since 2015/2016 improved again.

Github is github.
Then one has bitbucket, assembla, etc...

Bitbucket, huh? I've heard that term as a synonym for /dev/null (UNIX/Linux/BSD/OS X) or NUL: on PC- or MS-DOS/Windows systems. Personally, I don't like Sourceforge as much as I did when I established my two (or is that three?) projects there. However, it still fulfils a function. I now tend to use github, although now they've been bought by Microsoft... hm.


(Post 273)

[Sylvain Cote]

(08-31-2018 10:52 PM)brickviking Wrote:  I now tend to use github, although now they've been bought by Microsoft... hm.

50K projects moved from github to gitlab the week following the buyout announcement.

[Claudio L.]

(08-31-2018 10:52 PM)brickviking Wrote:  I thought that x49gp used the "usb" interface on the machine to connect to its 50g instance? Or can't I connect to it with ckermit after all? Maybe that's what's been crashing my machine every time I try it.

I think you confused x49gp with newRPL Desktop for a second. x49gp does not emulate USB (would be a nightmare to get it working, as it heavily depends on timings that are next to impossible to emulate properly).

[Claudio L.]

For all people following this thread: I committed changes to my repository that improve SD card emulation, now works with newRPL.

[brickviking]

Except now it doesn't work for me at all. I can't get a hp50g to "see" any sdcard in the filer regardless of whether I use a card image (dd if=...) or a folder. I open up the filer, and I don't see SDCARD, just ports 0-2 and Home. Darn.

Steps to duplicate:

• create 64Mb sdcard image (previously discussed): mkdosfs ....
• put sdcard image into .x49gp/sdcard
• start up x49gp
• right-click on screen, bring up the context menu
• select "Mount SD image..."
• Choose sdcard from filepicker, click Open
• turn on calculator if not already on
• open Filer [LS]-[G]
  

So, where do I go from here?

(Post 299 - I can almost feel it)

[3298]

I'm answering some comments from the NewRPL thread over here because I think they fit here better. Hope you don't mind.

(10-05-2018 06:01 AM)brickviking Wrote:  I'm guessing there's no way to "import" the USB-handling stuff from newRPL into x49gp, is there? That might just handle the one remaining issue left for x49gp's usefulness in comparison to the Connectivity Kit.

A few posts above (in this thread, not the one the question came from) Claudio has said it "would be a nightmare to get it working, as it heavily depends on timings that are next to impossible to emulate properly". He's the expert there, so I'll better stay away from that part of the code and avoid frustration. (Note that he didn't say "impossible" though - normal modern QEMU has some sort of USB passthrough for at least some of the systems it emulates, so it's technically possible; but unlike us those guys have a lot of real experts.)

(10-05-2018 06:01 AM)brickviking Wrote:  Also, I note that x49gp has a rather binary understanding of the 49g+/50G that doesn't take the newrpl UI into account. NewRPL's UI (at least in the desktop simulator) has modified several buttons, and I was wondering how easy it would be to have that implemented in some manner. Quite some time ago, there used to be a "image=..." parameter, but I note this has since been removed, citing lack of use or function. I think I just came up with a use: newRPL's UI.

Switching the image wouldn't have got you far, because the image doesn't contain the text labels. (There used to be a bug preventing the labels from showing up, which was worked around using an image with text, but that's been fixed.) And I suspect that the labels are exactly what you want to change.
I think the best way to support a NewRPL-adapted UI in x49gp would be to add variants of the existing "hp49g+" and "hp50g" models which use the same base image and the same bootcode as the existing ones, but with modified copies of the label definitions. That would actually be pretty easy: some light modifications to the existing model checks, and altered copies of the "static const x49gp_ui_key_t x49gp_ui_keys[]" and "static const x49gp_ui_key_t x50g_ui_keys[]" arrays (the difference between these is just a little bit of color - on the 49g+, the left-shift and right-shift keys have white labels, on the 50g they are black).
With a picture of the NewRPL Simulator in front of me (haven't had a reason to install it yet), I should be able to do this in a couple of minutes.

As a matter of fact, that's also a significant part of the code modifications needed for the suggested 48gII/39g+/39gs/40gs support. if somebody supplies me with text-less images and as well as bootcode dumps of these calculators (firmware would be nice too; I found firmware updates for 39gs and 40gs, but not for 39g+ and 48gII), I could probably do the rest of the work. These have different dimensions (both physical (including screen pixels) and memory-wise), but that's not a problem. For accurate emulation I'd also have to disable the SD slot for them (easy, but not sure if I'd actually want to do that, because it isolates them from the outside world due to a lack of I/O port emulation). The different sets of available I/O ports are something that would have to be taken care of if/when someone implements them.
Oh, by the way: a text-less image of a blue 50g would be nice too. Just to have the full set of models, you know.

(10-05-2018 08:55 AM)brickviking Wrote:  Except now it doesn't work for me at all. I can't get a hp50g to "see" any sdcard in the filer regardless of whether I use a card image (dd if=...) or a folder. I open up the filer, and I don't see SDCARD, just ports 0-2 and Home. Darn.

Again, Claudio is probably the person who is best equipped to fix this (I'd need to learn about these low-level SD commands first), but I suspect he inadvertently broke some of the SD commands the stock firmware uses, causing it to not detect the SD card anymore.

[Claudio L.]

Sorry, it seems even more changes were needed. Try it now.

[brickviking]

Yup, that worked. I tested with an image and a folder, just to make sure. Thanks for that, Claudio.

(Post 300!!)