New "spaceless" alternative mode where whitespace is opt-in with `{#+`.

[donquixote]

Current situation

Currently, adding a - to a twig tag causes whitespace removal.
E.g. {#- some comment #}, here whitespace is removed before but not after the comment.

Without the -, a single whitespace between twig tags is removed, but a double line break \n\n is not.

Proposal

I would like to propose a new mechanism that turns this around.

We introduce a new mode-switching twig tag {% spaceremoved %}, which would typically be placed at the top of a template.
(I would call it "spaceless" but that term now has a history. Open to suggestions.)

This allows to opt-in to this mechanism per template or per snippet, thus avoiding BC problems.

When this mode is enabled:

• A twig tag without - is treated as if it had -. E.g. {# some comment #} is treated like {#- some comment #}.
• A twig tag with + is treated like a regular tag without -. So, {#+ some comment +#} is treated like {# some comment #}.

Example

{% spaceremoved %} 
hel {# remove space before and after #} lo {#+ keep space before #} world

This is equivalent to

hel {#- remove space before and after -#} lo {# keep space before -#} world

which prints as

hello world

See also

I found

• Short spaceless-like operator #4442, which discusses the same problem, but does not propose this specific solution.

[donquixote]

We introduce a new mode-switching twig tag {% spaceremoved %}

I am not aware of any "mode-switching tag" that exists in twig, so this could be the challenging part.

[Seldaek]

https://twig.symfony.com/doc/3.x/tags/autoescape.html is kind of a mode-switching tag like you say.

[stof]

Whitespace control happens at the lexer level. Having a tag affecting the way the lexer operate is quite hard as this cannot be implemented using the tag parsing system.

Auto-escaping is about switching a runtime mode.

[ericmorand]

@stof

Whitespace control happens at the lexer level.

Which is wrong and should have been fixed years ago. - I'm sure it was mentioned already at least by myself. Your lexer is lossy. It is impossible to recreate the original code from the list of tokens. And as you now admit, it prevents from doing legitimate things at the AST level.

Saying that something is "quite hard" because of an issue that was created deliberately is, well, fallacious at best. The proposal makes sense, and should be straightforward to implement.

[stof]

The lexer being lossy is not an issue for Twig at all. Being able to recreate the original code from the tokens is not something Twig needs.

And we cannot do this trimming at the AST level. The AST is abstract, as indicated by its name, and does not have the notion of formatting anymore. Doing the whitespace control at parsing level (before producing the AST) would make the implementation a lot harder than today (and be a big BC break AFAICT, due to the extensible parser). Thus, it is also hard to implement given that whitespace control has the effect of changing the way the preceding or following text token is handled, changing its content. This happens in source order, not according to some AST structure.

Thus, if each node parser becomes responsible of whitespace control at the AST level, it means that whitespace control would not be supported for custom Twig tags unless their parser supports it. The reason why whitespace control is totally transparent for token parsers is precisely because it happens at the lexer level.

[donquixote]

Introducing a new mode in the lexer and supporting '+' in addition to '-' and '~' would indeed increase complexity. But it is not impossible.

I suppose we could still do some whitespace removal at a later stage, if we know where the twig tags are, and we have markers for the mode switching. Even in the AST we could walk through the tree and remove whitespace.
(I have only looked superficially so I could be wrong)
But if we want to support '+' as in {%+ ... +%}, this would be a lot more difficult.

If we do this, it should be in the lexer.
It's a tradeoff how popular or desirable this feature is vs the added complexity.

[donquixote]

Alternatively, to do it after the lexer:

• The lexer, when it finds a "+" in e.g. {%+, yields a new token "preserve whitespace", before the {% part.
• Something else processes the token stream and removes whitespace and the "preserve whitespace" tags according to the rules in the issue summary.

[donquixote]

If we do the same for "-" and "~", this would make that first stage of the lexer almost reversible / not lossy :)
Only that -%}{% would be indistinguishable from %}{%-. Unless we use distinct tokens for left/before vs right/after whitespace control, then it would be fully lossless.

(not saying that being lossy is a problem, just to make @ericmorand happy.)