Conversion from block mark-up to/from Markdown

[vadimkantorov]

Feature Request

• Yes, I reviewed the contribution guidelines.

Describe your use case and the problem you are facing

Export from existing WordPress installation's Gutenberg pages to Markdown for interop with other content engines or to LLMs which tend to like Markdown. Import would also be great. My usecase is to convert to Markdown the block markup syntax strings found in WXR files exported from an existing WP installation.

Describe the solution you'd like

Versatile / configurable export from block markup to Markdown. Custom attributes can be optionally exported via CommonMark Directive syntax which already has support in remark. Even if conversion is not 100% faithful, it is already useful: e.g. for making existing WordPress content consumable in good form for RAG / LLMs.

There are a lot of various existing markdown plugins for WordPress, but there is a need for a well-tested official one (which will be maintained, currently maintenance is also fragmented, there are plenty of abandoned exporters), but versatile one which can be scripted, so it's a good fit for CLI command, in my opinion.

Ideally, there must be handlers for standard block types, and some catch-all for optionally exporting verbatim HTML for the blocks which do not have registered markdown converter. Having some HTML in Markdown is also fine, as the user can then postproc the markdown files with remark/rehype pipelines.

IMO such exporter should accept that there does not exist a single right way to export to Markdown so some configurability is to be expected, as users may want to get different flavors of Markdown (e.g. both preserving the text color as Directives, or just dropping it altogether are good and useful, depending on the user's usecase).

Related:

• WP-CLI Command: convert ideas#187

For now, I had to roll my own https://github.com/vadimkantorov/wxr for exporting block markup from WXR to HTML files (preserving attributes), and then for converting them to Markdown (and also allowing some custom URL / image URL postprocessing / downloading the linked images etc). In this converter I'm often representing unknown blocks as:

---
blah blah some info about a block without a standard way to represent in Markdown (not frontmatter)
---

more markdown

So essentially using a pair of --- dividers to enclose some block info. Then a custom remark/rehype pipeline could interpret / render these in a custom way if a user desires so.

[swissspidy]

Some initial questions to understand this better:

While the title mentions a bidirectional conversion, your mostly interested in conversion to Markdown, is that correct?

Custom attributes can be optionally exported via CommonMark Directive syntax which already has support in remark.

What custom attributes would you have in mind here and why would they need to be preserved like that? Is it for the AI or because you'd like a lossless conversion?

Even if conversion is not 100% faithful, it is already useful: e.g. for making existing WordPress content consumable in good form for RAG / LLMs.

What would be your use case? Feeding post content from WP-CLI to a tool such as Claude Code?

There are a lot of various existing markdown plugins for WordPress, but there is a need for a well-tested official one (which will be maintained, currently maintenance is also fragmented, there are plenty of abandoned exporters), but versatile one which can be scripted, so it's a good fit for CLI command, in my opinion.

Which ones have you tried? Would be good to have an overview.

For now, I had to roll my own vadimkantorov/wxr for exporting block markup from WXR to HTML files

Are you looking for a WXR -> Markdown conversion or arbitrary block markup -> Markdown conversion? Those are two very different things.

---

My first thoughts off the top of my head:

For block markup -> Markdown conversion, this new https://github.com/dmsnell/html-to-md library is promising, as it powers the Markdown conversion currently enabled on WordPress.org. See e.g. https://make.wordpress.org/ai/?output_format=md for a demo.

For Markdown -> block conversion, I think the best route would be to get proper server-side block transforms into Gutenberg, see WordPress/gutenberg#13163 and https://github.com/chubes4/html-to-blocks-converter. Then, this can be turned into a Markdown -> HTML -> block conversion. It's an extra step, but it will be more precise.

Regarding different flavors and custom attributes, that certainly would increase scope and complexity. I'd probably just start with one.

[vadimkantorov]

While the title mentions a bidirectional conversion, your mostly interested in conversion to Markdown, is that correct?

Myself I am currently mainly interested of export to markdown. But I noticed that there also plenty of pluings for batch-ingesting Markdown content into WP: e.g. I've used postmark previously for this end: https://github.com/vadimkantorov/wpssg

What custom attributes would you have in mind here and why would they need to be preserved like that? Is it for the AI or because you'd like a lossless conversion?

E.g. in the basic case: text color, or aligning text right/left/center. For exporting to Markdown for LLM consumption these attributes matter less, but for the goal of transferring the Markdown content to other content engines (like Astro), it is important to have some options on ignoring / preserving these attributes

What would be your use case? Feeding post content from WP-CLI to a tool such as Claude Code?

My usecase currently is transferring existing WP posts to Astro engine, and turning them into Markdown to benefit from existing Markdown CMS / editors (such usecase requires a higher level of visual/semantic export faithfulness, compared to the LLM-only usecase). My other usecase would be building a client-side search-engine / RAG LLM search for my website, for that I'd like to also like to first convert all articles to Markdown and then make them available as llms.txt (which typically represents/links to Markdown content)

Which ones have you tried? Would be good to have an overview.

I have searched, found many (some free, some paid, many abandoned like postmark), but eventually decided that it would be simpler and more controlled way for me to manually convert block markup from WXR export, e.g. to have a simple custom way to post-process URLs (convert absolute URLs to relative by stripping a prefix and maybe prepending another prefix) and download some images locally.

Are you looking for a WXR -> Markdown conversion or arbitrary block markup -> Markdown conversion? Those are two very different things.

I did not care about WXR initially, but then I found that WXR contains block markup for all site's pages/posts, so I decided it is a good input to obtain all block markup archive. I care to get markdown for all pages/posts in an existing website. But if needed to manually first export WXR and then pipe it into a CLI/wp-cli tool, I think it's a suitable way. It's also good because it does not require being able to connect to a live MySQL server which can be not easy for some cheap hosted WordPress installations (often they do not offer a way to even connect to the database, or it is complex to establish such connections). So being able to run such conversion just from WXR without access to an actual WP installation (and being able to connect to WP database) is very valuable, that's why I think wp-cli command is also a preferrable to a WP plugin, as wp-cli can sometimes be used/installed even without live WP installation.