newRPL: Comments revisited

[Claudio L.]

The topic of comments in the code was touched here before, but never put formally:
Comments will work this way (subject to opinions):

• A single @ will start a comment, which ends with another @ or a newline
• A double @@ will start a permanent comment, which ends with @@ or a newline
• A triple @@@ will start a multiline comment, which only ends with @@@ (so it can be used to comment out lines that include normal and permanent comments).
• Comments cannot be nested.
• When a comment ends in a newline, the newline gets stored as part of the comment, so it gets restored when the comment gets decompiled.
  

[David Hayden]

What is the purpose of having the both single and double comments? They appear to do the same thing.

[rprosperi]

(11-02-2015 08:46 PM)Claudio L. Wrote:  The topic of comments in the code was touched here before, but never put formally:
Comments will work this way (subject to opinions):

• A single @ will start a comment, which ends with another @ or a newline
• A double @@ will start a permanent comment, which ends with @@ or a newline
• A triple @@@ will start a multiline comment, which only ends with @@@ (so it can be used to comment out lines that include normal and permanent comments).
• Comments cannot be nested.
• When a comment ends in a newline, the newline gets stored as part of the comment, so it gets restored when the comment gets decompiled.
  

These are flexible and cover just about any case one may need.

But maybe it's worth clarifying what is meant by "permanent comment". I've assumed this means said comment remains in the compiled object, but if not, then I agree with David, its confusing.

[Claudio L.]

(11-03-2015 01:56 AM)rprosperi Wrote:  But maybe it's worth clarifying what is meant by "permanent comment". I've assumed this means said comment remains in the compiled object, but if not, then I agree with David, its confusing.

Sorry, it makes no sense unless I explain it.
All comments will be compiled and will remain in the program, so when you decompile you get commented code.
However, there's a performance penalty (minimal) during execution (just skip the comment), so it's understandable that some people might want to strip the comments before releasing their code. There will be a command (STRIPCOMMENTS perhaps) that will do this task on already compiled code. There will also be a flag that controls whether comments are embedded or not in the compiled code (by default, comments will be preserved).
The STRIPCOMMENTS command and the flag will affect the comments as follows:

• Single line comments '@' will be removed by STRIPCOMMENTS, and won't be included in the code if the flag indicates so.
• Multi line comments '@@@' will be removed by STRIPCOMMENTS, and won't be included in the code if the flag indicates so.
• Single line permanent comments '@@' will be NOT be removed by STRIPCOMMENTS, and will always be included in the code regardless of flag settings.
  

The permanent comments are intended for copyright notices at the beginning of the code for example, that the programmer doesn't want removed automatically (they can still be removed manually by editing the code, but that would indicate malicious behavior of the user).

[matthiaspaul]

This is somewhat different from what I assumed. I assumed only @@ comments would end up in the compiled object, whereas @ and @@@ comments would be removed immediately - and therefore just useful when using an external editor. According to your description it depends on a special flag.

Is there a length limit for comment lines?

How are the comments stored internally? Is it a header indicating the type of comment and its size, so it can be "skipped", or are the @, @@, @@@ characters actually stored as part of the comment and thus wasting additional space?

Are comments patchable from outside (that is, no checksums etc.)? This would be desirable to implant version control info, serial numbers, perhaps even fixup data. Is it even possible to change the size of an embedded comment in a binary without invalidating the compiled object?

If comments are patchable, is the system safe when the number of newline characters in the comment changes through patching?

Is it possible for the RPL system to somehow retrieve data from inside comments? If so, build date or version info could be extracted from comments and processed or displayed (instead of putting this in some specially crafted variables).

Greetings,

Matthias

[rprosperi]

(11-03-2015 06:07 PM)Claudio L. Wrote:  Sorry, it makes no sense unless I explain it...

Thanks for clarifying this. I hadn't considered such a rich set of capabilities, but this sounds pretty thorough and useful.

(11-03-2015 10:01 PM)matthiaspaul Wrote:  This is somewhat different from what I assumed. I assumed only @@ comments would end up in the compiled object, whereas @ and @@@ comments would be removed immediately - and therefore just useful when using an external editor. According to your description it depends on a special flag.

Is there a length limit for comment lines?

How are the comments stored internally? Is it a header indicating the type of comment and its size, so it can be "skipped", or are the @, @@, @@@ characters actually stored as part of the comment and thus wasting additional space?

Are comments patchable from outside (that is, no checksums etc.)? This would be desirable to implant version control info, serial numbers, perhaps even fixup data. Is it even possible to change the size of an embedded comment in a binary without invalidating the compiled object?

If comments are patchable, is the system safe when the number of newline characters in the comment changes through patching?

Is it possible for the RPL system to somehow retrieve data from inside comments? If so, build date or version info could be extracted from comments and processed or displayed (instead of putting this in some specially crafted variables).

Greetings,

Matthias

Matthias raised some interesting questions and proposed some interesting capabilities; maybe more than planned (or needed) but nice ideas.

I think the most important point here is that the contents (or absence) of all comments (including the @ separators) should *not* be included in the checksum, so BYTES returns the same value of a given program, regardless of what is within the comments.

[Claudio L.]

Wow, never thought something like comments would require so much thinking...

(11-03-2015 10:01 PM)matthiaspaul Wrote:  This is somewhat different from what I assumed. I assumed only @@ comments would end up in the compiled object, whereas @ and @@@ comments would be removed immediately - and therefore just useful when using an external editor. According to your description it depends on a special flag.

Correct, the whole idea is to make comments more persistent for in-calc editing. On an external editor comments are the same as they've always been.

(11-03-2015 10:01 PM)matthiaspaul Wrote:  Is there a length limit for comment lines?

No limit. Same as strings.

(11-03-2015 10:01 PM)matthiaspaul Wrote:  How are the comments stored internally? Is it a header indicating the type of comment and its size, so it can be "skipped", or are the @, @@, @@@ characters actually stored as part of the comment and thus wasting additional space?

Just a string with a different prolog.
Right now, the way it's implemented is that the prolog is the same for all comments. Then the second and possibly the third @ symbol are stored as part of the text (the first one isn't). The ending characters are always stored as part of the text (be it a newline or an @ symbol) to preserve the original formatting.

(11-03-2015 10:01 PM)matthiaspaul Wrote:  Are comments patchable from outside (that is, no checksums etc.)? This would be desirable to implant version control info, serial numbers, perhaps even fixup data. Is it even possible to change the size of an embedded comment in a binary without invalidating the compiled object?

newRPL does not use checksums (yet). So yes, in theory you could write self-commenting code, but I don't see any reason to create RPL commands to do such a thing. You'd have to use the low-level functions available from C, but it's possible.

(11-03-2015 10:01 PM)matthiaspaul Wrote:  If comments are patchable, is the system safe when the number of newline characters in the comment changes through patching?

Why would the number of newline characters compromise the system? I can see a million ways to crash the system if you try to patch the comments in a binary object, but adding newlines won't necessarily do it.
Patching comments can only be done through low-level access, and once you are doing low-level things, not only you can patch a comment, you can also overwrite the whole object, and write outside the object, and the stack, and the whole RAM, even erase the Flash and brick your calculator.
But not from within RPL, you'd have to write a library in C to do that.

(11-03-2015 10:01 PM)matthiaspaul Wrote:  Is it possible for the RPL system to somehow retrieve data from inside comments? If so, build date or version info could be extracted from comments and processed or displayed (instead of putting this in some specially crafted variables).

All you have to do is decompile the object and analyze the resulting source code to detect those specially crafted comments and do something useful. Doxygen comes to mind, as the best example of using specially crafted comment lines to do something good.
Same thing works for "patching" comments by decompiling the code, patching the text and recompiling it. This is the correct RPL way of doing it, since objects should all be copy-on-write, no in-place patching should be allowed.

[sa-penguin]

If I want text to stay in code (copyright statement), I typically create a string constant.

Anything else ends up being stripped, to optimise size and / or speed.

[matthiaspaul]

(11-04-2015 03:21 AM)Claudio L. Wrote:  Wow, never thought something like comments would require so much thinking...

Well, you asked for comments... ;-)

(11-04-2015 03:21 AM)Claudio L. Wrote:  

(11-03-2015 10:01 PM)matthiaspaul Wrote:  How are the comments stored internally? Is it a header indicating the type of comment and its size, so it can be "skipped", or are the @, @@, @@@ characters actually stored as part of the comment and thus wasting additional space?

Just a string with a different prolog.
Right now, the way it's implemented is that the prolog is the same for all comments. Then the second and possibly the third @ symbol are stored as part of the text (the first one isn't). The ending characters are always stored as part of the text (be it a newline or an @ symbol) to preserve the original formatting.

Okay, sounds good as it is, but still leaves some tiny potential for further improvement.

(11-04-2015 03:21 AM)Claudio L. Wrote:  

(11-03-2015 10:01 PM)matthiaspaul Wrote:  Are comments patchable from outside (that is, no checksums etc.)? This would be desirable to implant version control info, serial numbers, perhaps even fixup data. Is it even possible to change the size of an embedded comment in a binary without invalidating the compiled object?

newRPL does not use checksums (yet). So yes, in theory you could write self-commenting code, but I don't see any reason to create RPL commands to do such a thing. You'd have to use the low-level functions available from C, but it's possible.

(11-04-2015 03:21 AM)Claudio L. Wrote:  

(11-03-2015 10:01 PM)matthiaspaul Wrote:  If comments are patchable, is the system safe when the number of newline characters in the comment changes through patching?

Why would the number of newline characters compromise the system? I can see a million ways to crash the system if you try to patch the comments in a binary object, but adding newlines won't necessarily do it.

Well, before your answer (above) I had no insight into your implementation. Since you mentioned that newlines at the end of a comment are a special case, I can see potential implementations which would be breakable by inserting newlines into a comment. Since you asked for comments on your specification, I was just going through my mental checklist of things that could go wrong or could get into the way of future expansion, and wanted to make sure, that they are non-issues in your implementation...

Quote:Patching comments can only be done through low-level access

I was thinking about stuff like libraries with embedded comments in a cross-development environment, where f.e. a version control system and PC tools might be used to patch files. In-system patching might, perhaps, be useful for tools like runtime profilers or debuggers (to store some kind of "metadata" in comments).

Greetings,

Matthias

[Claudio L.]

(11-04-2015 04:31 PM)matthiaspaul Wrote:  Well, before your answer (above) I had no insight into your implementation. Since you mentioned that newlines at the end of a comment are a special case, I can see potential implementations which would be breakable by inserting newlines into a comment. Since you asked for comments on your specification, I was just going through my mental checklist of things that could go wrong or could get into the way of future expansion, and wanted to make sure, that they are non-issues in your implementation...

You are right, the way it was implemented (as a universal string object for all comment types) leaves no room for crashes due to newlines (hence you left me scratching my head wondering how that could be), but perhaps if I had coded a different kind of object for the single liners and you added a newline it wouldn't be allowed.
But the current implementation only distinguishes the type of comment during compile time, to determine if a newline ends the comment or not. After compile is finished, a comment is a comment, nothing can break it.

(11-04-2015 04:31 PM)matthiaspaul Wrote:  I was thinking about stuff like libraries with embedded comments in a cross-development environment, where f.e. a version control system and PC tools might be used to patch files. In-system patching might, perhaps, be useful for tools like runtime profilers or debuggers (to store some kind of "metadata" in comments).

Yes, although we could have a dedicated metadata object too... so comments remain just comments, and true metadata cannot be removed with STRIPCOMMENTS. I wrote a post about a structured data object a short while back that didn't get any attention, but that's probably where metadata belongs, in some sort of opaque LIBDATA object.

[compsystems]

Hey, why not use the copyright symbol ©

3 cases
© comment the whole line
©/ .... \© interval one line
©* *© block or multiline

code
3 4 +
5 6 *
7 8 ^



a: comment the whole line
3 4 + © Comments ...
5 6 *
7 8 ^

b: interval
3 4 + ©/ 5 6 \© 7 8 ^

c: multiline
3 4 +
©*
5 6 *
7 8 ^
*©

or
3 4 + ©* 5 6 7 8 ^ *©



I also wish separator instructions
semicolon symbol

3 4 +;
5 6 *;
7 8 ^;

=>

3 4 +; 5 6 *; 7 8 ^;

[BarryMead]

(11-17-2015 07:44 PM)compsystems Wrote:  Hey, why not use the copyright symbol ©

I can think of one reason. The Copyright symbol is NOT available for direct typing
on the keyboard (At least not on my 104 key US keyboard). Therefore it would
reduce the ease of entering comments (forcing the user cut and paste the symbol
from a reference file or an internet web page showing all symbols).

I think it is wise to make it as EASY AS POSSIBLE for programmers to add comments
as they are often already too lazy to put in sufficient documentation. Programmers don't need
any more hurdles to jump over or any more excuses to provide less documentation.

In my humble opinion :-)

[Claudio L.]

(11-17-2015 07:44 PM)compsystems Wrote:  Hey, why not use the copyright symbol ©

There's a few reasons:
a) The @ symbol is part of the original RPL specification, so it makes sense to keep it for compatibility.
b) It's common to include a copyright message in programs, so we need the copyright symbol to mean just that.
c) As Barry mentioned, not easy to type if you are coding on a PC (we could make it available on the calculator keyboard).

(11-17-2015 07:44 PM)compsystems Wrote:  3 cases
© comment the whole line
©/ .... \© interval one line
©* *© block or multiline

@ comment the whole line
@ .... @ interval one line
@@@ @@@ block or multiline

I think your idea is very close to what's implemented right now except for the symbol (but even the symbol looks similar), so we are thinking along the same lines.

[timofonic]

Any news about this project?

What about merging with WP43S?