Methods for inserting HTML should be exposed on `OD_Template_Optimization_Context` for before or after all tags have been visited

[westonruter]

Originally discussed in #1919 (comment) and in other comments on that PR.

The OD_HTML_Tag_Processor class includes two methods: ::append_head_html() and ::append_body_html(). The OD_HTML_Tag_Processor instance is exposed on an OD_Tag_Visitor_Context instance passed to the tag visitors, but it is not yet exposed on the OD_Template_Optimization_Context class which is passed to the od_start_template_optimization and od_finish_template_optimization actions which run before and after the document has been iterated over by tag visitors, respectively.

The current use cases for ::append_head_html() are:

• Optimization Detective injecting the LINK HTML markup returned by OD_Link_Collection::get_html().
• Image Prioritizer adding a STYLE tag via a tag visitor to add a style for lazy loaded background images.
• Embed Optimizer adding a STYLE tag via a tag visitor to reduce layout shifts.
• Content Visibility adding a STYLE tag via a tag visitor for CV styles.

The current use cases for ::append_body_html() are:

• Optimization Detective uses this to insert the detect.js script to the page.
• Embed Optimizer adding a SCRIPT to the end of the BODY when there is a lazy-loaded embed on the page.
• Image Prioritizer adding a SCRIPT tag via a tag visitor to lazy load background images.
• Image Prioritizer adding a SCRIPT tag via a tag visitor to lazy load videos.

Allowing insertion of HTML once via the od_finish_template_optimization avoids the need for tag visitors to keep track of whether they inserted or not. They can use the tag visitor callbacks to get a "lay of the land" by looking at all of the tags, and then at the od_finish_template_optimization action they can finalize what they need to insert in the HEAD or the BODY.

This could, for example, allow tag visitors to better optimize stylesheets they insert into the document. Instead of Embed Optimizer inserting a separate STYLE for each embed to reserve space to reduce layout shifts, it could instead insert a single STYLE at od_finish_template_optimization which combines all the style rules in one stylesheet. For example, this would allow Embed Optimizer to better group styles by media query instead of having to output @media (width <= 480px) {} for each embed on the page. Currently for each embed it inserts a STYLE like:

<style>
@media (width <= 480px) { #embed-optimizer-6040306707fb51ccaafa915c2da8f412 { min-height: 500px; } }
@media (480px < width <= 600px) { #embed-optimizer-6040306707fb51ccaafa915c2da8f412 { min-height: 500px; } }
@media (600px < width <= 782px) { #embed-optimizer-6040306707fb51ccaafa915c2da8f412 { min-height: 500px; } }
@media (782px < width) { #embed-optimizer-6040306707fb51ccaafa915c2da8f412 { min-height: 500px; } }
</style>
<style>
@media (width <= 480px) { #embed-optimizer-96ffd32b51748c70b288af1ef0c14c01 { min-height: 500px; } }
@media (480px < width <= 600px) { #embed-optimizer-96ffd32b51748c70b288af1ef0c14c01 { min-height: 500px; } }
@media (600px < width <= 782px) { #embed-optimizer-96ffd32b51748c70b288af1ef0c14c01 { min-height: 500px; } }
@media (782px < width) { #embed-optimizer-96ffd32b51748c70b288af1ef0c14c01 { min-height: 500px; } }
</style>
<style>
@media (width <= 480px) { #embed-optimizer-f9ff6c9a914366ac3c1aab1994dd8a69 { min-height: 500px; } }
@media (480px < width <= 600px) { #embed-optimizer-f9ff6c9a914366ac3c1aab1994dd8a69 { min-height: 500px; } }
@media (600px < width <= 782px) { #embed-optimizer-f9ff6c9a914366ac3c1aab1994dd8a69 { min-height: 500px; } }
@media (782px < width) { #embed-optimizer-f9ff6c9a914366ac3c1aab1994dd8a69 { min-height: 500px; } }
</style>

With the od_finish_template_optimization action, it could just print the @media at-rules once for each viewport group rather than duplicating them, and output them all in a single STYLE tag rather than in three separate ones, for example:

<style>
@media (width <= 480px) {
	#embed-optimizer-6040306707fb51ccaafa915c2da8f412 { min-height: 500px; }
	#embed-optimizer-96ffd32b51748c70b288af1ef0c14c01 { min-height: 500px; }
	#embed-optimizer-f9ff6c9a914366ac3c1aab1994dd8a69 { min-height: 500px; }
}
@media (480px < width <= 600px) {
	#embed-optimizer-6040306707fb51ccaafa915c2da8f412 { min-height: 500px; }
	#embed-optimizer-96ffd32b51748c70b288af1ef0c14c01 { min-height: 500px; }
	#embed-optimizer-f9ff6c9a914366ac3c1aab1994dd8a69 { min-height: 500px; }
}
@media (600px < width <= 782px) {
	#embed-optimizer-6040306707fb51ccaafa915c2da8f412 { min-height: 500px; }
	#embed-optimizer-96ffd32b51748c70b288af1ef0c14c01 { min-height: 500px; }
	#embed-optimizer-f9ff6c9a914366ac3c1aab1994dd8a69 { min-height: 500px; }
}
@media (782px < width) {
	#embed-optimizer-6040306707fb51ccaafa915c2da8f412 { min-height: 500px; }
	#embed-optimizer-96ffd32b51748c70b288af1ef0c14c01 { min-height: 500px; }
	#embed-optimizer-f9ff6c9a914366ac3c1aab1994dd8a69 { min-height: 500px; }
}
</style>

See #1923 for a proof of concept for this.

Additionally, for tag visitors that insert scripts at the end of the BODY based on whether certain tags are encountered, this could be done via the od_finish_template_optimization action instead. Otherwise the tag visitor needs to keep track with a added_lazy_script class member variable for whether it inserted the script yet, although the difference here is not as great as with the stylesheets in Embed Optimizer. Example: #1922.

Being able to insert HTML via these actions would also be useful to a plugin like Optimization Detective Admin UI which needs to insert the current response's URL Metric data for rendering in the admin bar. See westonruter/od-admin-ui#11 for an example of how this could be implemented.

Currently when tag visitors call ::append_head_html() or ::append_body_html(), they must be sure to pass raw HTML which is valid in the HEAD and BODY context respectively. If they don't, then they'll cause a parsing error for the browser (e.g. adding an IMG in the HEAD will prematurely open the BODY).

Optimization Detective could continue to use the ::append_head_html() and ::append_body_html(), but for extensions there would need to be new methods like:

• ::append_head_style() to append an inline stylesheet
• ::append_head_script() (but also differentiate between inline vs non-inline?)
• ::append_body_script() (ditto)

We wouldn't need to include an ::append_head_link() since this is what the OD_Link_Collection takes care of, but we may need to add support for rel=stylesheet.

Other potential methods that will be useful:

• ::append_body_stylesheet_link() (not used yet, but useful for adding non-blocking external stylesheets)
• ::append_head_meta() to add a META tag
• ::append_body_comment() to add a comment to the end of the BODY
• ::append_document_comment() to add a comment after the closing </html> tag.

Note that we should perhaps not allow script modules to be inserted in the HEAD because import map scripts (SCRIPT[type="importmap"]) are printed in the footer in Classic Themes, and if a script module appears before an import map then it prevents the import map from working.

If these methods were exposed on both OD_Tag_Visitor_Context and OD_Template_Optimization_Context then the ::append_html_html() and ::append_body_html() methods could be deprecated and eventually not exposed at all.

See also the following from #1546:

We should consider not directly exposing the OD_HTML_Tag_Processor to the tag visitors and template optimization start/finish action handlers since this then exposes the append_head_html() and append_body_html() methods which may not be desirable. This also exposes a lower-level API than which may be appropriate which will complicate switching between using WP_HTML_Tag_Processor and WP_HTML_Processor implementations. Namely, for WP_HTML_Processor there may not be a need to add a get_xpath() method directly to the class, and we may not need to subclass it at all. If we instead passed a wrapper class, then we could expose a get_xpath() on the wrapper subclass which is then able to use the appropriate implementation based on whether it is using WP_HTML_Tag_Processor or WP_HTML_Processor. In fact, this could allow Optimization Detective to use an unsubclassed original WP_HTML_Tag_Processor instance which is provided by core for output-buffer manipulation instead of allowing filtering the underlying string (if that is so desired, per Core-43258).

[felixarntz]

@westonruter Thanks for opening this, and for the detailed description.

I like having methods for specific use-cases rather than allowing any HTML. Not only does this prevent problematic usage, but it also encourages extensions to use these new methods where possible, rather than continuing to use the tag visitor context's tag processor's broader (but therefore also more complicated-to-use) HTML insertion methods.

Some additional thoughts and questions:

• Instead of having methods prefixed with append_head_, append_body_, append_document_, how about we instead introduce some constants / an enum-like class or interface with constants that represents these "locations"? That way we could have fewer methods, and they could be called like e.g. append_style( Location::HEAD, /* stylesheet */ ).
• I'm not sure yet on how we should best name the methods when it comes to inline vs external stylesheets/scripts. If we want to lean on Core's naming, anything just called "style" or "script" would be considered external / not inline, and "inline style" and "inline script" would denote the inline variants. Alternatively, we could use explicit method names where we always refer to "external style" / "external script" / "inline style" / "inline script". The one approach I'm not a fan of is to rely on the "link" term - while technically that's used for an "external style", I think it's unnecessarily low level for the API's intended usage.
• On the PR we discussed potentially requiring an identifier for any such asset added, which would avoid the need for the extension to check whether a specific style or script was already added or not. The underlying implementation could have that baked-in. Alternatively, we could potentially solve it internally even without identifiers, where arguably we could just take the actual script or style string and not inject it if it was already injected, since there's no value in injecting the exact same CSS or JS - in JS there may be theoretical usages like this, but not sure whether that's relevant to consider here.
• It might be a good idea to add these methods on a newly introduced class that wraps the (private) OD_HTML_Tag_Processor instance. An instance of the new class (naming ideas??) could then be made available on OD_Template_Optimization_Context via __get(). This would be in line with how extensions access the tag processor on OD_Tag_Visitor_Context. Eventual usage would be like: $template_context->{new thing}->append_....
• Are there any remaining use-cases where tag visitors need to be able to modify HTML via their own tag visitor callback? Of course they need to retain access to the OD_HTML_Tag_Processor instance to read attributes etc., but I wonder whether we should somehow lock down (or abstract away) the ability to inject arbitrary HTML from the tag visitors as well, to further clarify and harden the API.

Curious to get your thoughts.

[westonruter]

@felixarntz

• Instead of having methods prefixed with append_head_, append_body_, append_document_, how about we instead introduce some constants / an enum-like class or interface with constants that represents these "locations"? That way we could have fewer methods, and they could be called like e.g. append_style( Location::HEAD, /* stylesheet */ ).

Thinking about this some more, I think we can actually be a bit more opinionated about where the HTML is inserted. For example, OD_Link_Collection provides an interface to add links but it doesn't let you control where they're added.

For example, when inserting a script module, it should always be appended to the BODY because otherwise if it is inserted before an import map, this can cause an error. So we can have these methods:

• add_script_module( string $src, array $attrs = array( 'async' => false ) )
• add_inline_script_module( string $text, array $attrs = array( 'async' => false ) )

There's no need to be able to specify defer since they are deferred by default. Also, there's no use case currently for non-inline script modules:

performance/plugins/image-prioritizer/class-image-prioritizer-video-tag-visitor.php

Line 244 in 26442b9

$processor->append_body_html( wp_get_inline_script_tag( image_prioritizer_get_video_lazy_load_script(), array( 'type' => 'module' ) ) );

performance/plugins/image-prioritizer/class-image-prioritizer-background-image-styled-tag-visitor.php

Line 237 in 26442b9

$processor->append_body_html( wp_get_inline_script_tag( image_prioritizer_get_lazy_load_bg_image_script(), array( 'type' => 'module' ) ) );

performance/plugins/embed-optimizer/class-embed-optimizer-tag-visitor.php

Line 331 in 26442b9

$processor->append_body_html( wp_get_inline_script_tag( embed_optimizer_get_lazy_load_script(), array( 'type' => 'module' ) ) );

And there aren't any needs so far to insert a non-module script, so perhaps we just not consider these for now.

For STYLE tags, they should generally always be added to the HEAD. The only exception to this is styles for keyframes, which AMP requires to be placed at the end of the BODY. So we could have:

• add_style( $text ) (or add_inline_style( $text )) which always appends to the HEAD
• add_keyframes_style( $text ) when needed, which always appends to the BODY

But we can just go with the first for now.

• Alternatively, we could use explicit method names where we always refer to "external style" / "external script" / "inline style" / "inline script". The one approach I'm not a fan of is to rely on the "link" term - while technically that's used for an "external style", I think it's unnecessarily low level for the API's intended usage.

That said, there is the OD_Link_Collection which could be used for adding stylesheets. This would allow the stylesheets to be loaded even earlier via Link HTTP response headers.

• Alternatively, we could potentially solve it internally even without identifiers, where arguably we could just take the actual script or style string and not inject it if it was already injected, since there's no value in injecting the exact same CSS or JS - in JS there may be theoretical usages like this, but not sure whether that's relevant to consider here.

Yeah, that makes sense to me.

• It might be a good idea to add these methods on a newly introduced class that wraps the (private) OD_HTML_Tag_Processor instance. An instance of the new class (naming ideas??) could then be made available on OD_Template_Optimization_Context via __get(). This would be in line with how extensions access the tag processor on OD_Tag_Visitor_Context. Eventual usage would be like: $template_context->{new thing}->append_....

Yes, this is what I was getting at with what I quoted above from #1546.

The one hiccup here is that in the context of the tag visitor, all of the methods on OD_HTML_Tag_Processor (subclassing WP_HTML_Tag_Processor) would need to be available:

• ::add_class()
• ::change_parsing_namespace() (not sure how this is used)
• ::class_list()
• ::expects_closer()
• ::get_attribute()
• ::get_attribute_names_with_prefix()
• ::get_breadcrumbs()
• ::get_comment_type()
• ::get_current_depth()
• ::get_doctype_info()
• ::get_full_comment_text()
• ::get_modifiable_text()
• ::get_namespace()
• ::get_qualified_attribute_name()
• ::get_qualified_tag_name()
• ::get_tag()
• ::get_token_name()
• ::get_token_type()
• ::get_updated_html() (this need not be exposed)
• ::get_xpath()
• ::has_bookmark()
• ::has_class()
• ::has_self_closing_flag()
• ::is_tag_closer()
• ::next_tag()
• ::next_token()
• ::release_bookmark()
• ::remove_attribute()
• ::remove_class()
• ::seek()
• ::set_attribute()
• ::set_bookmark() (we can prevent tag visitors here from setting reserved bookmark names)
• ::set_meta_attribute()
• ::set_modifiable_text()
• ::subdivide_text_appropriately() (not sure why this would be needed)

However, none of these should be exposed on OD_Template_Optimization_Context since the processor is either not yet started or it is already completed walking over the document.

So I think we'd continue to provide OD_Tag_Visitor_Context::$processor but it wouldn't be direct access to the OD_HTML_Tag_Processor but rather a wrapper which could switch between WP_HTML_Tag_Processor or WP_HTML_Processor in the future. This wrapper could add additional restraints about what methods are allowed to be called. For example, tag visitors must not be able to set or clear internal bookmark names:

• optimization_detective_end_of_head
• optimization_detective_end_of_body
• optimization_detective_current_tag

What OD_Template_Optimization_Context and OD_Tag_Visitor_Context both need is an additional property for handling the insertion of new HTML into the document. For example, it could be it could be an OD_HTML_Document_Inserter class which is exposed as OD_Template_Optimization_Context::$document_inserter and OD_Tag_Visitor_Context::$document_inserter which then exposes these methods:

• add_inline_style()
• add_inline_script_module()

Or these methods could be defined right on OD_Template_Optimization_Context and OD_Tag_Visitor_Context which then call the underlying method on the underlying HTML Processor implementation (which are not present on the processor wrapper exposed on OD_Tag_Visitor_Context).

• Are there any remaining use-cases where tag visitors need to be able to modify HTML via their own tag visitor callback? Of course they need to retain access to the OD_HTML_Tag_Processor instance to read attributes etc., but I wonder whether we should somehow lock down (or abstract away) the ability to inject arbitrary HTML from the tag visitors as well, to further clarify and harden the API.

I think the above handles all the cases currently identified for extensions. Optimization Detective itself would need to be able to insert raw HTML, of course, to be able to insert the link tags, but this needn't be exposed to the extensions.

[westonruter]

As part of this, the optimization loop should directly have access to WP_HTML_Tag_Processor::set_attribute() and not the subclassed version so that it can avoid adding the data-od- meta attributes as seen in #1954. For example, it is currently resulting in:

<meta data-od-replaced-content="optimization-detective 0.0.0" name="generator" content="optimization-detective 0.0.0; url_metric_groups={0: empty, 480: empty, 600: empty, 782: empty}">

When it should instead output:

<meta name="generator" content="optimization-detective 0.0.0; url_metric_groups={0: empty, 480: empty, 600: empty, 782: empty}">

The OD meta attributes are intended for tag visitors to show what changes they made on the document.

[felixarntz]

@westonruter

Thinking about this some more, I think we can actually be a bit more opinionated about where the HTML is inserted. For example, OD_Link_Collection provides an interface to add links but it doesn't let you control where they're added.

For example, when inserting a script module, it should always be appended to the BODY because otherwise if it is inserted before an import map, this can cause an error.

I like that idea, let's start with being opinionated and only supporting script modules, not regular scripts. We could always add support for the latter later, if there proves to be a need.

For STYLE tags, they should generally always be added to the HEAD. The only exception to this is styles for keyframes, which AMP requires to be placed at the end of the BODY. So we could have:

• add_style( $text ) (or add_inline_style( $text )) which always appends to the HEAD
• add_keyframes_style( $text ) when needed, which always appends to the BODY

But we can just go with the first for now.

+1, I think the keyframes requirement is too specific to consider for now. Also, AMP requiring it doesn't necessarily make a compelling argument for supporting it here, since it's neither officially supported by Core nor a standard of some kind.

• Alternatively, we could use explicit method names where we always refer to "external style" / "external script" / "inline style" / "inline script". The one approach I'm not a fan of is to rely on the "link" term - while technically that's used for an "external style", I think it's unnecessarily low level for the API's intended usage.

That said, there is the OD_Link_Collection which could be used for adding stylesheets. This would allow the stylesheets to be loaded even earlier via Link HTTP response headers.

Wouldn't that only allow for external stylesheets? In any case, even if we use OD_Link_Collection under the hood, I think the API would be more intuitive if we exposed consistent methods on our new class for script modules and stylesheets.

The one hiccup here is that in the context of the tag visitor, all of the methods on OD_HTML_Tag_Processor (subclassing WP_HTML_Tag_Processor) would need to be available:

[...]

However, none of these should be exposed on OD_Template_Optimization_Context since the processor is either not yet started or it is already completed walking over the document.

I was thinking that the new class would only be responsible for writing/adding stuff to the HTML output, not reading. I think (at least for now), the tag visitor context could maintain access to the OD_HTML_Tag_Processor instance to get information about the HTML.

So I think we'd continue to provide OD_Tag_Visitor_Context::$processor but it wouldn't be direct access to the OD_HTML_Tag_Processor but rather a wrapper which could switch between WP_HTML_Tag_Processor or WP_HTML_Processor in the future. This wrapper could add additional restraints about what methods are allowed to be called. For example, tag visitors must not be able to set or clear internal bookmark names:

• optimization_detective_end_of_head
• optimization_detective_end_of_body
• optimization_detective_current_tag

Per what I answered above, I think this is worth thinking about, but I think as part of this issue we should limit the scope to focusing on the write interactions only via the new class.

What OD_Template_Optimization_Context and OD_Tag_Visitor_Context both need is an additional property for handling the insertion of new HTML into the document. For example, it could be it could be an OD_HTML_Document_Inserter class which is exposed as OD_Template_Optimization_Context::$document_inserter and OD_Tag_Visitor_Context::$document_inserter which then exposes these methods:

• add_inline_style()
• add_inline_script_module()

See my previous reply, that's roughly what I meant for the new class to have as scope.

I think the above handles all the cases currently identified for extensions. Optimization Detective itself would need to be able to insert raw HTML, of course, to be able to insert the link tags, but this needn't be exposed to the extensions.

👍

[westonruter]

@felixarntz

Wouldn't that only allow for external stylesheets? In any case, even if we use OD_Link_Collection under the hood, I think the API would be more intuitive if we exposed consistent methods on our new class for script modules and stylesheets.

That's right, it would just be for external stylesheets. But considering that one performance pattern is to put stylesheets in the footer (e.g. if they are intended to not be blocking) then this doesn't really fit with OD_Link_Collection. So yeah, probably a separate method like ::add_external_style() should be added (eventually). This would be in addition to ::add_inline_style() for inline styles.

I was thinking that the new class would only be responsible for writing/adding stuff to the HTML output, not reading. I think (at least for now), the tag visitor context could maintain access to the OD_HTML_Tag_Processor instance to get information about the HTML.

If they had access to the OD_HTML_Tag_Processor then they'd also have access to the ::append_head_html() and ::append_body_html() methods. And otherwise, WP_HTML_Tag_Processor (and WP_HTML_Processor) have many other methods for writing HTML, like ::set_attribute() and ::set_modifiable_text(), so it would seem strange to be able to use the processor instance for some writes but then this other HTML writer class for other writes. I think better to add that abstraction now so from the tag visitor's perspective (and the perspective of od_start_template_optimization and od_finish_template_optimization) for them to still have access to the one processor which exposes an interface which contains the desired methods from WP_HTML_Tag_Processor/WP_HTML_Processor with the additional augmented methods for special HTML insertions. This makes sense now as well prior to a 1.0.0 stable version since it is impacting the primary interface used by extensions.