Intenal Object format

[devin122]

Has any work been done on determining the internal object format that the Prime uses? I started poking around for fun, but I thought I'd check to make sure I wasn't wasting my time on something that has already been solved.

[devin122]

I'll take the silence to mean no. So for the record here is what I have so far. Note that there is the in-memory layout and the serialized layout on disk. Differences are pointed out where relevant. Data is stored on disk in little-endian format

The basic layout for all objects is:

Code:


struct THPObj {
  uint16_t reference_count; 
  // definition of flag bits varies with type,
  // but the least significant bit indicates a heap allocated object 
  uint8_t flags:4;
  ObjTag tag:4; // Indicates the type of this object
};

enum ObjTag {
  TAG_REAL = 0,
  TAG_INT = 1,
  TAG_STRING = 2,
  TAG_COMPLEX = 3,
  TAG_MATRIX = 4,
  TAG_ERROR = 5,
  TAG_LIST = 6,
  TAG_IDENT = 7,
  TAG_FUNC_CALL = 8,
  TAG_UNIT = 9,
  TAG_INSTRUCTION_SEQUENCE = 10, // Just a list with special semantics
  TAG_USERFUNC = 11,
  TAG_LIST_PROCESSOR = 12,
  TAG_EVALUATOR_REQUEST = 13,
  TAG_GEN = 14 // Wrapper around a giac::gen object

For clarity, since bitfield ordering is non-standard, the flags occupy the high 4 bits of the 3rd byte, while the tag occupy the low 4 bits.

In the serialized format, reference count values of 0xFFFF and 0xFFFE sometimes act as special indicators, but I haven't figured them all out yet.
Reals:

Code:


struct Real: public THPObj{
  int8_t sign_stuff;
  int32_t exponent;
  uint64_t mantissa; // Packed BCD
};

Flag of 2 causes number to be displayed as DMS.
For sign_stuff 0 = NaN, 1 = normal, 2 = +Inf, -1 = -normal, and -2 = -Inf.
Also it lookes like only the top 56 bits of the mantissa are looked at, but I haven't done much digging.
Ints:

Code:


struct Integer: public THPObj {
  int8_t num bits; // rage of [-64,64]. Negative values indicated a signed value
  uint8_t padding[4];
  uint64_t data;
};

Flag bits appear to be used to change the displayed base of the number, but I haven't bothered mapping them all out yet.

Strings:

Code:


struct String: public THPObj {
  uint8_t padding;
  TSize num_chars;
  TChar data[];
};

In memory TChar = char32_t, while on disk it is char16_t.
On disk if the ref count = 0xFFFF, or in memory, TSize = uint32_t.
On disk if the ref count != 0xFFFF, TSize =uint16_t.

Func Calls:
This info applies to in-memory format only I haven't looked too much at the serialized format

Code:


struct FunctionCall: public THPObj {
  uint8_t param_count;
  uint8_t padding[4]; // Probably only on 64-bit, but haven't verified.
  THPObj* unknown_obj;
  void* definition; // pointer to some kind of data structure which describes the function.
  THPObj* params[];
};

Note that in the serialized format list of parameter pointers is replaced with the serialized parameters.

[jfelten]

where did you get this information?

[rprosperi]

Unlike all previous HP models, Prime was designed from the get-go for use in schools systems and includes systemic features built-in (Exam mode) to preserve the integrity of the system, also allowing the instructor to control which features are/are not available to the student when in restricted modes. Not providing free memory access, "peek" like instructions, details on internals, etc. was probably done in that spirit, to help preserve said system integrity.

Doing things to make it easier for users to hack the system and possibly circumvent these safeguards could be met with even more draconian limitations and/or system changes to 'fight back'. Exam mode integrity seems to be of paramount importance to the success of the Prime. Given the limited resources that seem to be available, it would be a shame if they had to waste effort in those areas.

So, while I'd never (ever!) suggest you should not explore and document this stuff, I can say you should not be surprised if the current details and access methods change as a result.

Just something to think about.

[devin122]

(08-02-2021 05:15 PM)jfelten Wrote:  where did you get this information?

Most of it was obtained by reverse engineering the firmware / using AFiles() to save and reload objects. A couple of the more obscure tag values I got from Cyrilles HPCC presentation

(08-02-2021 10:43 PM)rprosperi Wrote:  So, while I'd never (ever!) suggest you should not explore and document this stuff, I can say you should not be surprised if the current details and access methods change as a result.

I have very much been keeping that in mind, as I would hate to see more restrictions put on the calculator. In fact, I have picked up some details about exam mode which I have no intention of sharing for the reason that it would really not serve any purpose other than defeating the exam mode. This info however I felt might be interesting to some people, and the risk of abuse seems relatively low. Plus, in order to use this info for nefarious purposes would require a level of skill which means you could figure it out on your own.

[rprosperi]

(08-04-2021 06:45 PM)devin122 Wrote:  

(08-02-2021 05:15 PM)jfelten Wrote:  where did you get this information?

Most of it was obtained by reverse engineering the firmware / using AFiles() to save and reload objects. A couple of the more obscure tag values I got from Cyrilles HPCC presentation

I wondered at the time if anyone was examining the code seen in screen while he was showing some stuff. Good eyes!

(08-04-2021 06:45 PM)devin122 Wrote:  

(08-02-2021 10:43 PM)rprosperi Wrote:  So, while I'd never (ever!) suggest you should not explore and document this stuff, I can say you should not be surprised if the current details and access methods change as a result.

I have very much been keeping that in mind, as I would hate to see more restrictions put on the calculator. In fact, I have picked up some details about exam mode which I have no intention of sharing for the reason that it would really not serve any purpose other than defeating the exam mode. This info however I felt might be interesting to some people, and the risk of abuse seems relatively low. Plus, in order to use this info for nefarious purposes would require a level of skill which means you could figure it out on your own.

Got it. As you say, most folks that can use such info have no need to cheat Exam Mode, but it is probably best to not simply post such info, as those who do need it, may well have associates who could use it....

Thanks.

[Tim Wessman]

(08-04-2021 06:45 PM)devin122 Wrote:  I have very much been keeping that in mind, as I would hate to see more restrictions put on the calculator. In fact, I have picked up some details about exam mode which I have no intention of sharing for the reason that it would really not serve any purpose other than defeating the exam mode. This info however I felt might be interesting to some people, and the risk of abuse seems relatively low. Plus, in order to use this info for nefarious purposes would require a level of skill which means you could figure it out on your own.

Thanks.

Most object types are quite straight forward. If there is any specific info you want go ahead and ask and if can will provide. However looks like you already are able to get what you want pretty easily so far so...