I implemented a custom pipeline -- here are my observations

[julia-and]

Hello,

First, thank you for building Statiq. I like the "framework" approach as it really enables one to use static site generation perfectly tailored to each use case.

I'm used to Hugo's system for defining menus, but I've been watching for a SSG that would let me use Razor for templates. So when I found Statiq, I figured I'd learn it by building a pipeline to provide similar menu functionality.

It took a while to figure out what I was doing, but I now have a prototype working. Here are my observations.

General

1. The pipeline system is very powerful and flexible once you understand it. Nice!
2. Flattening/discarding non-scalar front matter data should be optional. There are a lot of uses for structured metadata. Objects defined in YAML front matter are ignored and I don't see any way to disable that. Objects seem to work in JSON side-car files. I had to create a custom ParseStructuredYaml module.
3. No support for JSON front matter? Not a big deal for me, but in my testing it seems as simple as
   new ExtractFrontMatter("+++", new ParseJson())? (the +++ delimiter is how Hugo delimits TOML front matter, but whatever).
4. Not as fast as Hugo, but doesn't need to be. We'll be using this in an automated deployment scenario obviously. I've only tested with a relatively small amount of mixed content so far.
5. Razor breakpoints don't work in Jetbrains Rider. (Probably a Rider issue...? works fine in Visual Studio).

Documentation

1. The documentation about pipelines and modules on statiq.dev could use expansion in a few areas:
   • Include a visual of the relationships/dependencies between the built-in pipelines in Statiq.Web
   • Discuss how output from a custom pipeline will be handled by built-in pipelines when using DependencyOf. (ex, the Content and Data pipelines Concat() documents output by their dependencies, but DirectoryMetadata does not appear to. I had to read the code to figure that out)
   • Discuss how built-in modules use the caching mechanism. For the menu module I have to read all content files... it would be nice if I could utilize the cache so the Content pipeline doesn't have to read everything again. (Currently, I have simply replicated the modules in the Content pipeline through CacheDocuments() to try and achieve this).
2. I read a lot of the code to learn how things work. The code is pretty well-commented... an API reference site would have helped me learn about the different built-in modules and what they do.

Questions

1. Currently, some pipelines like Content will Concat() outputs from their dependencies, but what if instead of adding documents I want to modify them before they are rendered? Is it kosher to subclass the built-in Content pipeline and insert new processing modules? (If the answer is yes... it would be nice to see this in the documentation!)
2. IExecutionContext.Settings is typed as an IReadOnlyConfigurationSettings, but it actually returns an IConfigurationSettings, which has an Add() method exposed. (I used this to put the menu data I extract from front matter and config files into the global metadata.) But if it's "read only", should it really have an Add() method?
3. Is there a better way to make some metadata globally available for use during rendering?

Hope this helps!

[daveaglick]

This feedback is invaluable, thanks so much for taking the time to get it all down!

Flattening/discarding non-scalar front matter data should be optional.

Totally agree. The ParseYaml module actually has an alternate constructor that adds the YAML values as dynamic, including the full object graph, but I don't love that. Ideally I think nested non-scaler values should be represented as IMetadata. How did you implement your ParseStructuredYaml module?

No support for JSON front matter?

It's been on my todo list for Statiq Web for a while to add JSON and XML front matter support. The challenge (if you can call it that - I don't actually think it'll be very hard) is figuring out how to indicate which front matter parser to use (or ideally figure it out based on content, like an initial { means JSON, < means XML, and anything else means YAML). I don't love the convention of different delimiters for different kinds of front matter, but could support that too depending on how widespread it's become.

Not as fast as Hugo

Totally agree - a lot of the performance lag right now is due to the Razor rendering engine and there's not a lot we can do about that. It's just not designed for fast-throughput compilation and rendering. What we can do though is provide alternate, more generator friendly view engines. I'm hope to continue adding more of those (a PR to add Scriban support is in the works already). Then it's a tradeoff of power (Razor) vs. speed (Handlebars, Liquid, etc.) and the user can choose the appropriate approach.

Razor breakpoints...works fine in Visual Studio

I don't know much about Rider, but this seriously works in VS? I've never tried - that's so cool!

The documentation about pipelines and modules on statiq.dev could use expansion in a few areas

Excellent feedback, I'll work on incorporating more documentation in these areas.

an API reference site would have helped me learn about the different built-in modules and what they do

Agreed, myself included. I'm planning on doing something very similar to the Wyam docs once Statiq Docs is far enough along to do code analysis on the codebase and extract module classes: https://wyam.io/modules/

but what if instead of adding documents I want to modify them before they are rendered

Great question - this could definitely be addressed better in the docs. The best way of doing it is to modify the pipeline to add your modules. For example, I recently had a use case where I wanted to add geocoding to documents that contained a metadata key of Location so I wrote a little custom module and added it to the pipeline like this (on the Bootstrapper):

...
.ConfigureEngine(x => x
    .Pipelines[nameof(Statiq.Web.Pipelines.Content)]
    .ProcessModules
    .Add(new GeocodeLocations()))

Pretty sure I'll also add a new .ModifyPipeline() method to the bootstrapper in the next release to make this pattern a little clearer. The key is that IEngine.Pipelines is essentially a dictionary of pipelines and each pipeline is essentially a list of modules. They can be manipulated as such until execution.

IExecutionContext.Settings is typed as an IReadOnlyConfigurationSettings, but it actually returns an IConfigurationSettings

I actually just finished refactoring the whole settings support in the most recent release, though the new code works mostly like the old code in practice. This is actually a convention several execution context properties follow: a mutable object that's exposed as such in the engine (presumably for use prior to execution) that also implements a read-only interface that gets presented in the context. The intent is that you'll mostly interact with the object via the exposed read-only interface - though as with any interface you can also use reflection and casting to get the underlying object even if it's not intended.

Is there a better way to make some metadata globally available for use during rendering?

I'd try to avoid manipulating global metadata during execution. Because pipelines run concurrently, modifying anything globally during execution can end up being non-deterministic depending on ordering and that can create hard-to-find bugs and discrepancies. Instead I'd create a thread-safe singleton to store and provide that kind of data - you could either get it via a static method or by registering with the DI container as a singleton service.

Thanks again! And don't hesitate to let me know if you have any other questions or feedback.

[julia-and]

How did you implement your ParseStructuredYaml module?

My ParseStructuredYaml module is pretty basic. It just deserializes as a dictionary and sticks it in the metadata... it's easier to work with than dynamic.

protected override IEnumerable<IDocument> ExecuteInput(IDocument         input,
				        		IExecutionContext context)
{
	var documentMetadata = new List<Dictionary<string, object>>();
	
	using (TextReader contentReader = new StreamReader(input.GetContentStream()))
	{
		var metaData = _deserializer.Deserialize<Dictionary<string, object>>(contentReader);
		documentMetadata.Add(metaData);
	}

	return documentMetadata.Select(metadata => input.Clone(metadata));
}

I agree with your thought that parsing non-scalar YAML values would be best represented as IMetadata.

an initial { means JSON, < means XML, and anything else means YAML

Yes, I definitely like that better than delimiting with +++, ---, etc.
Some people like TOML data, too, although that would be difficult to tell apart from Yaml by an initial character.

a lot of the performance lag right now is due to the Razor rendering engine

I figured... but it's a worthwhile trade-off for the power of Razor when you need it!

Pretty sure I'll also add a new .ModifyPipeline() method to the bootstrapper

That makes sense, thanks!

I'd try to avoid manipulating global metadata during execution.

Yes, I've since found it's better to output the aggregated menu data to a single document that can be referenced later from the execution context Outputs.FromPipeline() collection.

Thanks again for your hard work and support!

[julia-and]

Forgot to mention, I believe it would make sense to have a setting to determine whether or not to discard non-scalar metadata.

[Perksey]

Found myself implementing something like ParseStructuredYaml in my pipeline - would love to have something like this in the framework.

I'd be willing to PR it, but is the ParseYaml/ParseStructuredYaml something that would be desirable? Or perhaps maybe something like a YamlFlattenMode which will either do the current logic or just directly merge the deserialized YAML Dictionary<string, object> into the metadata dictionary?

[julia-and]

Personally, I would consider it very useful. I do like the idea of a YamlFlattenMode...