Get the end_line and end_col of a multi-line comment?

[PeterPanino]

While the uses node:
<USES begin_line="7" begin_col="1" end_line="14" end_col="1">
... as a multiline element has the attributes begin_line, begin_col, end_line, end_col,
the Comments-node object has only the members line and col. Although a comment can be multiple lines.

So how can I determine the end_line and end_col of a multi-line comment?

[RomanYankovsky]

But can't you calculate this from comment text?

[PeterPanino]

Yes, but this would be a kind of guessing. Having an EXPLICIT syntax tree information about the end_line and end_col of a multi-line comment would be better.

[bogdanpolak]

@RomanYankovsky I think that this yet another to decide which way to go: close (out of scope) or triage it as a new feature: great idea is to assign labels: enhancement and help wanted to highlight your decision

[JackTrapper]

Most Abstrat Syntax Tree parsers traditionally do not put comments in the tree.

More modern AST parsers are realizing that we want to keep everything from the source code; especially and including comments.

Herb Sutter talked about in this work to modernize the C++ language the way Bjarne did:

I stick close to what the original indentation was, including that I keep track of the location of all comments and put them right back where I found them.

And when HTML5 is parsed into a tree, it keeps comments in the tree.

But SimpleParser (and the DelphiAST wrapper around it) does not maintain comments in the AST.

Issue #292 does note that comments are preserved, in a separate list. But they are not maintained in-place; which means you cannot do things like what Python and Lua do: allow extra notation to be added to the language through comments, and then analyzed by a tool. E.g.:

The simplest example is adding a comment annotation that indicates that a reference variable could be nullable using a syntax of C#:

• Before: customer: TCustomer;
• After: customer: TCustomer{?}; //customer variable cannot be nil

// @requiresPositive(CustomerID)
function GetCustomer(CustomerID: Integer): TCustomer{?};
begin
    if CustomerID <= 0 then
    begin
       Result := nil;
       Exit;
    end;

Now the comment is actually part of the syntax tree:

ntComment Value="@requiresPositive(CustomerID)"
ntMethod @anKind="function" @anName="GetCustomer"
├─ ntParameters 
 │  ╰─ ntParameter 
 │     ├─ ntName Value="CustomerID" 		(TValuedSyntaxNode)
 │     ╰─ ntType @anName="Integer"
 ├─ ntReturnType 
 │  ├─ ntType @anName="TCustomer"
 │  ╰─ ntComment Value="?"
 ╰─ ntStatements 		(TCompoundSyntaxNode)

But this is not the fault of SimpleParser.pas; which was originally written in 1999. This is a more modern developement.

And along with more robust parsing and being more resilient to errors (like HTML5) and never leaving the tree in an undefined state. If the syntax is wrong, all Delphi parsers should generate the same tree. From HTML5:

This specification defines the parsing rules for [HTML documents], whether they are syntactically correct or not. Certain points in the parsing algorithm are said to be parse errors. The error handling for parse errors is well-defined (that's the processing rules described throughout this specification)

Ideally parse errors would be well-handled:

One can build parsers with built-in error recovery. There are many papers on this sort of thing, a hot topic in the 1980s. Check out Google Scholar, hunt for "syntax error repair". The basic idea is that the parser, on encountering a parsing error, skips to some well-known beacon (";" a statement delimiter is pretty popular for C-like languages, which is why you got asked in a comment if your language has statement terminators), or proposes various input stream deletions or insertions to climb over the point of the syntax error. The sheer variety of such schemes is surprising. The key idea is generally to take into account as much information around the point of error as possible. One of the most intriguing ideas I ever saw had two parsers, one running N tokens ahead of the other, looking for syntax-error land-mines, and the second parser being feed error repairs based on the N tokens available before it encounters the syntax error. This lets the second parser choose to act differently before arriving at the syntax error. If you don't have this, most parser throw away left context and thus lose the ability to repair