41CL image identifiers

[Jurgen Keller]

As I can't remember the image identifiers of the 400+ modules the 41CL provides, I often have to look them up in the manual. Today I decided to implement a faster way for searching image identifiers. The ad hoc plan was to convert the table in the manual into a CSV file and somehow create a searchable HTML page out of it.

Extracting tables from PDF files can be a nightmare. Luckily there is a tool that does the job: Tabula. So after a few minutes I got the CSV file. After some unsuccessful attempts I also found a way to create a searchable HTML file out of it. The key was another tool: CSVtoTable.

The image identifier table in the 41CL manual uses colors to indicate on which hardware version the image is available. I decided to add another column to the CSV file that contains this information (HW). That was a bit of work and hopefully I didn't do any mistakes. The last problem was that I also wanted to search by group name. As CSVToTable always searches the full table, I simply added a dollar sign in front of the group names. Now, searching for, e.g., $GAM provides all the game related images.

I've published the HTML page on my web site, see:
41CL Image Identifiers

So the only problem left is to keep the table up to date.

[Sylvain Cote]

Hello Jurgen,

This is great, thank you!

CSV files (for v2, v3/4, v5) are available at 41CL Calculator : Manuals and listed as TXT files.
I would suggest to use these in your conversion process because Monte update them each time he release a new package version.

Sylvain

[rprosperi]

(04-13-2019 06:06 AM)Jurgen Keller Wrote:  I've published the HTML page on my web site, see:
41CL Image Identifiers

So the only problem left is to keep the table up to date.

Thanks Jurgen for creating and sharing this very useful resource!

I'll bet you're really, really glad Monte had reduced his update cadence to only once a month.

[Monte Dalrymple]

(04-13-2019 06:06 AM)Jurgen Keller Wrote:  As I can't remember the image identifiers of the 400+ modules the 41CL provides, I often have to look them up in the manual. Today I decided to implement a faster way for searching image identifiers. The ad hoc plan was to convert the table in the manual into a CSV file and somehow create a searchable HTML page out of it.

Extracting tables from PDF files can be a nightmare. Luckily there is a tool that does the job: Tabula. So after a few minutes I got the CSV file. After some unsuccessful attempts I also found a way to create a searchable HTML file out of it. The key was another tool: CSVtoTable.

The image identifier table in the 41CL manual uses colors to indicate on which hardware version the image is available. I decided to add another column to the CSV file that contains this information (HW). That was a bit of work and hopefully I didn't do any mistakes. The last problem was that I also wanted to search by group name. As CSVToTable always searches the full table, I simply added a dollar sign in front of the group names. Now, searching for, e.g., $GAM provides all the game related images.

I've published the HTML page on my web site, see:
41CL Image Identifiers

So the only problem left is to keep the table up to date.

Thank you for your contribution Jurgen!

If you are using the 41CL Extreme Functions (YFNX) then a similar feature is built into the IMDB? function.

To search for all mnemonics starting with a particular letter use the search pattern "Mxx?" where "M" is the starting letter and "xx" are don't-care.

To search for all mnemonics of a particular type use the search pattern "?TTT" where "TTT" is the image type.

Both of these searches can be stopped/restarted/single-stepped, and leave the current mnemonic in ALPHA for easy use with the PPLUG function.

[Jurgen Keller]

It looks like I missed several things, thank you all for your hints.

The TXT files might be the right starting point for an automated process. I'll have a closer look. I usually don't like to do repeated tasks, even if it's only once a month! :-) Which reminds me of Is It Worth the Time?

The IMDB? function looks indeed promising. I'm feeling a bit oldschool right now, I guess I have to switch to YFNX NOW.

[Ángel Martin]

(04-13-2019 04:23 PM)Jurgen Keller Wrote:  The IMDB? function looks indeed promising. I'm feeling a bit oldschool right now, I guess I have to switch to YFNX NOW.

Most definitely, YFNX is *the* way to use the CL without a doubt.
IMDB? is just one reason but there are many other advantages to it.

BTW I've been considering a re-design of the CLLIB and ROMLIB functions in the PowerCL_Extreme. Currently they use their own subset of the IMDB with the ROM id# mnemonics instead of looking in the IMDB itself, which is not the best approach for complete coverage and also requires constant maintenance. Besides, the space needed has already exceeded the available room in the module, so I need to find a better alternative.

IMDB? is almost perfect, but I'll like to find a faster way to scan the entries, something that CLLIB/ROMLIB do. We'll see if I can come up with a good idea...

Best,
ÁM

[Monte Dalrymple]

(04-14-2019 05:21 AM)Ángel Martin Wrote:  

(04-13-2019 04:23 PM)Jurgen Keller Wrote:  The IMDB? function looks indeed promising. I'm feeling a bit oldschool right now, I guess I have to switch to YFNX NOW.

Most definitely, YFNX is *the* way to use the CL without a doubt.
IMDB? is just one reason but there are many other advantages to it.

BTW I've been considering a re-design of the CLLIB and ROMLIB functions in the PowerCL_Extreme. Currently they use their own subset of the IMDB with the ROM id# mnemonics instead of looking in the IMDB itself, which is not the best approach for complete coverage and also requires constant maintenance. Besides, the space needed has already exceeded the available room in the module, so I need to find a better alternative.

IMDB? is almost perfect, but I'll like to find a faster way to scan the entries, something that CLLIB/ROMLIB do. We'll see if I can come up with a good idea...

Best,
ÁM

Just out of curiosity, how do CLLIB/ROMLIB scan faster? Perhaps I can implement a better algorithm. I know that removing the "SEARCHING" message would make things faster...

[Jurgen Keller]

Just wanted to let you know that I'm now able to update the 41CL Image Identifiers Online Table on a regular base. I've reduced the efforts for an update to a couple of minutes.

[Ángel Martin]

(05-10-2019 06:17 PM)Jurgen Keller Wrote:  Just wanted to let you know that I'm now able to update the 41CL Image Identifiers Online Table on a regular base. I've reduced the efforts for an update to a couple of minutes.

A very useful tool indeed, specially the sorting ability by category.

One suggestion (always the pesky users ;-) : Can you add a "Location" column with the flash address to the table?

Best,
ÁM

[Sylvain Cote]

(05-11-2019 05:03 AM)Ángel Martin Wrote:  

(05-10-2019 06:17 PM)Jurgen Keller Wrote:  Just wanted to let you know that I'm now able to update the 41CL Image Identifiers Online Table on a regular base. I've reduced the efforts for an update to a couple of minutes.

One suggestion (always the pesky users ;-) : Can you add a "Location" column with the flash address to the table?

Pesky users sometime comes in pair
If you add the location column, do not forget to add the following information:

• beta/v2 addresses
• v3/v4 addresses & v5 (first half of flash space)
• v5 (second half of flash space)
  

[Jurgen Keller]

Ángel, Sylvain,

Glad to hear that you like the image identifier table. I've just published a new version, I hope that the additional information is what you've asked for. Please note that maybe you'll have to reload the page to see the changes (at least that was the case with Safari).

[Monte Dalrymple]

Jurgen,

This is excellent! Much appreciated!

A couple of anomalies though:

1. For CMGT the XROM number appears in the "Restrictions" column.

2. For H67G the XROM number is "TBD" even though it's listed as 23 in mem_ref.

One comment is that when sorting by the XROM number the sort is not by
number, but by the first digit? (So that 2 is sorted after 19.) Any way to sort
by numeric value? (So the 2 is sorted after 1.)

I noticed that I am not consistent with listing multi-XROM cases (sometimes used "," to separate values.) I'll fix that with this month's
Flash update.

Monte

[Jurgen Keller]

(05-11-2019 07:31 PM)Monte Dalrymple Wrote:  A couple of anomalies though:
1. For CMGT the XROM number appears in the "Restrictions" column.

I've inherited this issue from the 41CL manual, see page 50.

(05-11-2019 07:31 PM)Monte Dalrymple Wrote:  2. For H67G the XROM number is "TBD" even though it's listed as 23 in mem_ref.

Same situation here, see 41CL manual page 52.

The table creation process works like this:
1. Extract image identifier table from 41CL manual into CSV
2. Enrich this information with hardware version and FLASH page read from memref_*.txt files
3. Convert CSV to HTML page using CSVtoTable

(05-11-2019 07:31 PM)Monte Dalrymple Wrote:  One comment is that when sorting by the XROM number the sort is not by
number, but by the first digit? (So that 2 is sorted after 19.) Any way to sort
by numeric value? (So the 2 is sorted after 1.)

This is indeed a problem, however, I currently cannot influence that because, as pointed out before, I don't create the HTML page by myself. Apparently CSVtoTable does currently treat all columns as texts.

I also explored to create the table from the memory reference files and the IMDB source files directly (not using the table from the manual at all). However, I stopped that. There were many small differences when comparing the output of both approaches.

[Jurgen Keller]

(05-11-2019 07:31 PM)Monte Dalrymple Wrote:  One comment is that when sorting by the XROM number the sort is not by
number, but by the first digit? (So that 2 is sorted after 19.) Any way to sort
by numeric value? (So the 2 is sorted after 1.)

I did some further research and I think the sorting problem will be solvable. Instead of using the CSVtoTable tool I'll consider to generate the HTML directly which allows some customization.

[Monte Dalrymple]

(05-11-2019 07:57 PM)Jurgen Keller Wrote:  

(05-11-2019 07:31 PM)Monte Dalrymple Wrote:  A couple of anomalies though:
1. For CMGT the XROM number appears in the "Restrictions" column.

I've inherited this issue from the 41CL manual, see page 50.

(05-11-2019 07:31 PM)Monte Dalrymple Wrote:  2. For H67G the XROM number is "TBD" even though it's listed as 23 in mem_ref.

Same situation here, see 41CL manual page 52.

The table creation process works like this:
1. Extract image identifier table from 41CL manual into CSV
2. Enrich this information with hardware version and FLASH page read from memref_*.txt files
3. Convert CSV to HTML page using CSVtoTable

(05-11-2019 07:31 PM)Monte Dalrymple Wrote:  One comment is that when sorting by the XROM number the sort is not by
number, but by the first digit? (So that 2 is sorted after 19.) Any way to sort
by numeric value? (So the 2 is sorted after 1.)

This is indeed a problem, however, I currently cannot influence that because, as pointed out before, I don't create the HTML page by myself. Apparently CSVtoTable does currently treat all columns as texts.

I also explored to create the table from the memory reference files and the IMDB source files directly (not using the table from the manual at all). However, I stopped that. There were many small differences when comparing the output of both approaches.

Now I feel like more of an idiot than usual (which is saying something.) I apologize, Jurgen. The manual has been fixed.

[Ken S]

The number order could probably be fixed if the xrom numbers less than 10 had a zero in front of them. It is a typical windows fault, if you have numbered file names you need the zeros as place holders.

[Jurgen Keller]

(05-11-2019 10:34 PM)Ken S Wrote:  The number order could probably be fixed if the xrom numbers less than 10 had a zero in front of them. It is a typical windows fault, if you have numbered file names you need the zeros as place holders.

I've thought about that, too. However, in the end I've found another solution that doesn't need artificial zeros. Thanks for the hint anyway.

[Jurgen Keller]

(05-11-2019 10:03 PM)Monte Dalrymple Wrote:  

(05-11-2019 07:57 PM)Jurgen Keller Wrote:  I've inherited this issue from the 41CL manual, see page 50.


Same situation here, see 41CL manual page 52.

The table creation process works like this:
1. Extract image identifier table from 41CL manual into CSV
2. Enrich this information with hardware version and FLASH page read from memref_*.txt files
3. Convert CSV to HTML page using CSVtoTable


This is indeed a problem, however, I currently cannot influence that because, as pointed out before, I don't create the HTML page by myself. Apparently CSVtoTable does currently treat all columns as texts.

I also explored to create the table from the memory reference files and the IMDB source files directly (not using the table from the manual at all). However, I stopped that. There were many small differences when comparing the output of both approaches.

Now I feel like more of an idiot than usual (which is saying something.) I apologize, Jurgen. The manual has been fixed.

I'm sorry, that was for sure not my intent. Four eyes see more than two. Thanks a lot for the update, I will take over the changes.

[Jurgen Keller]

I've now changed the process a bit and do not use CSVtoTable anymore but write the HTML page directly. I've published an updated HTML page:

• Fixed XROM sorting issue.
• Improved display of group identifiers. No more dollar sign in front, however, you may still use the dollar sign to filter by groups (e.g., search $GAM will show the images belonging to the game group).
• Improved display of Page column using sub-columns.
• Consistent display of XROM lists (commas always replaced by slashes)
• Multi-column sorting: shift click on a column to add clicked column as a secondary, tertiary etc. ordering column.
  

Known issues:

• In Safari the table title sometimes disappears when sorting the columns. I've checked with Firefox, no problem there. So I assume it's a short-coming in Safari. I didn't found a workaround yet.
  

[Sylvain Cote]

Thank you Jurgen for these nice updates.

(05-12-2019 12:06 PM)Jurgen Keller Wrote:  Known issues:

• In Safari the table title sometimes disappears when sorting the columns. I've checked with Firefox, no problem there. So I assume it's a short-coming in Safari. I didn't found a workaround yet.
  

I have done at least 30 refreshes on different columns and I am not experiencing this issue with Safari 12.1 on macOS High Sierra 10.13.6

Insignificant issue: the up/down arrow associated with the V2 sub-column is displayed on the side of the Page column.