Adding XML Support

CHANGELOG.md

@@ -5,6 +5,56 @@ All notable changes to the TOON specification will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [4.0] - 2026-01-15
+
+### Added
+
+- **XML Construct Support** (§22-28): TOON now natively supports XML constructs in a unified syntax:
+  - **Namespaces** (§23): Declare namespaces using `xmlns` and `xmlns:prefix` keys, consistent with XML namespace declaration syntax
+  - **Attributes** (§24): Represent XML attributes as nested key-value pairs within element objects
+  - **Repeated Elements** (§25): Use standard array syntax `key[N]:` for repeated XML elements, with support for interleaved elements using list form
+  - **Mixed Content** (§26): Represent mixed content (text and elements interleaved) using arrays where strings are text nodes and objects are elements
+  - **CDATA and Text Content** (§27): Handle CDATA sections, entity references, and whitespace normalization
+  - **XML Encoding/Decoding** (§28): Complete rules for converting between XML and TOON formats
+- **Extended Syntax Features** (§4.1):
+  - Namespace declarations via `xmlns` and `xmlns:prefix` keys
+  - Empty field names in tabular headers to represent text content (§26.4)
+  - Array-based syntax for elements with both attributes and text content (§24.3)
+- **XML-Related Terms** (§1.10): Added terminology for namespace prefixes, qualified names (QName), attributes, and mixed content
+- **XML Data Model** (§2.2): Extended data model to support XML infoset constructs
+- **XML Normalization** (§3.1): Rules for normalizing XML documents to TOON format
+- **Extended Syntax Errors** (§14.6): Strict-mode validation for undefined namespace prefixes
+- **XML Interoperability** (§17.4): Mapping guide for XML to TOON conversion
+- **XML Examples** (Appendix H): Comprehensive examples demonstrating XML constructs in TOON
+- **Encoder Options** (§13, §28.3):
+  - `preserveNamespaces` (boolean, default: true): Preserve namespace prefixes when encoding
+- **Decoder Options** (§28.4):
+  - Attribute values are always coerced to strings when decoding to XML (numeric and boolean values converted to string representations)
+- **Grammar Extensions** (§6): ABNF grammar updated to support namespace prefixes in keys (`ns-prefix` production)
+- **Key Encoding** (§7.3): Keys may include namespace prefixes following pattern `^([A-Za-z_][A-Za-z0-9_]*:)?[A-Za-z_][A-Za-z0-9_.]*$`
+- **Media Type Registration** (§18.1): Added optional `mode` parameter (`json` or `xml`) to `text/toon` media type (auto-detection applies if absent)
+
+### Changed
+
+- **Abstract** (§Abstract): Updated to reflect unified JSON and XML support
+- **Purpose and Scope** (§Introduction): Expanded to include XML document serialization use cases
+- **Data Model** (§2): Restructured into JSON Data Model (§2.1) and XML Data Model (§2.2)
+- **Type Terms** (§1.6): Updated object and array definitions to support XML constructs
+- **Conformance Checklists** (§13): Added XML-related requirements for encoders, decoders, and validators
+- **Strict Mode Errors** (§14): Added extended syntax error validation (undefined namespace prefixes)
+- **IANA Considerations** (§18): Updated media type registration to include XML mode
+
+### Removed
+
+- None (backward compatible with v3.0)
+
+### Notes
+
+- XML comments, processing instructions, and DOCTYPE declarations are not supported and are dropped during XML-to-TOON encoding
+- The default namespace applies to unprefixed element names but NOT to unprefixed attribute names (per XML Namespaces specification)
+- Built-in namespaces (`xml` and `xmlns`) are predefined and MUST NOT be redeclared
+- All XML support is additive and does not break existing JSON functionality
+
 ## [3.0] - 2025-11-24
 
 ### Breaking Changes

examples/README.md

@@ -88,6 +88,34 @@ Complete, valid TOON files demonstrating core features:
   - Decoder option `expandPaths="safe"` reconstructs nested structure
   - Spec: §13.4 Path Expansion
 
+### XML Constructs (v4.0+)
+
+- [`valid/xml-namespaces.toon`](valid/xml-namespaces.toon) - Namespaced document
+  - Demonstrates `xmlns` and `xmlns:prefix` namespace declarations
+  - Shows qualified names with namespace prefixes
+  - Spec: §23 Namespaces
+
+- [`valid/xml-attributes.toon`](valid/xml-attributes.toon) - XML attributes
+  - Demonstrates attributes as nested key-value pairs
+  - Shows attributes with and without namespace prefixes
+  - Shows elements with attributes and text content using arrays
+  - Spec: §24 Attributes
+
+- [`valid/xml-repeated-elements.toon`](valid/xml-repeated-elements.toon) - Interleaved repeated elements
+  - Demonstrates interleaved elements of different types (XML-specific feature)
+  - Shows how order is preserved when elements are mixed (not representable in JSON)
+  - Spec: §25.4 Preserving Order with Interleaved Elements
+
+- [`valid/xml-mixed-content.toon`](valid/xml-mixed-content.toon) - Mixed content
+  - Demonstrates mixed content with text nodes and elements
+  - Shows arrays where strings are text nodes and objects are elements
+  - Spec: §26 Mixed Content
+
+- [`valid/xml-xhtml.toon`](valid/xml-xhtml.toon) - XHTML example
+  - Complete XHTML document with namespaces and attributes
+  - Demonstrates complex XML document structure
+  - Spec: §22-28 XML Constructs
+
 ## Invalid Examples
 
 Examples that intentionally violate TOON syntax rules:
@@ -119,6 +147,11 @@ Examples that intentionally violate TOON syntax rules:
   - MUST error in strict mode
   - Spec: §6 Header Syntax (delimiter equality requirement)
 
+- [`invalid/xml-undefined-namespace.toon`](invalid/xml-undefined-namespace.toon) - Undefined namespace prefix (v4.0+)
+  - Uses prefix `ex` without corresponding `xmlns:ex` declaration
+  - MUST error in strict mode
+  - Spec: §14.6 Extended Syntax Errors, §23 Namespaces
+
 ## Conversions
 
 Side-by-side JSON ↔ TOON examples showing equivalent representations:
@@ -140,6 +173,22 @@ Side-by-side JSON ↔ TOON examples showing equivalent representations:
   - Shows practical use case for serializing API responses while preserving deterministic structure
   - `expandPaths="safe"` is not required for this file (no folded keys)
 
+- [`conversions/xml-catalog.json`](conversions/xml-catalog.json) + [`conversions/xml-catalog.toon`](conversions/xml-catalog.toon) (v4.0+)
+  - Library catalog with repeated book elements
+  - Demonstrates tabular format for uniform repeated XML elements
+  - Shows attributes and text content in XML structure
+  - Spec: §25 Repeated Elements
+
+- [`conversions/xml-rss.json`](conversions/xml-rss.json) + [`conversions/xml-rss.toon`](conversions/xml-rss.toon) (v4.0+)
+  - RSS feed example with attributes and repeated items
+  - Demonstrates tabular format for RSS item arrays
+  - Spec: §25 Repeated Elements
+
+- [`conversions/xml-interleaved.json`](conversions/xml-interleaved.json) + [`conversions/xml-interleaved.toon`](conversions/xml-interleaved.toon) (v4.0+)
+  - Interleaved repeated elements of different types
+  - Demonstrates list form for preserving element order
+  - Spec: §25.4 Preserving Order with Interleaved Elements
+
 ## Using These Examples
 
 These examples are useful for:

examples/conversions/xml-catalog.json

@@ -0,0 +1,21 @@
+{
+  "library": {
+    "book": [
+      {
+        "id": 1,
+        "title": "1984",
+        "author": "George Orwell"
+      },
+      {
+        "id": 2,
+        "title": "Brave New World",
+        "author": "Aldous Huxley"
+      },
+      {
+        "id": 3,
+        "title": "Fahrenheit 451",
+        "author": "Ray Bradbury"
+      }
+    ]
+  }
+}

examples/conversions/xml-catalog.toon

@@ -0,0 +1,5 @@
+library:
+  book[3]{id,title,author}:
+    1,1984,George Orwell
+    2,Brave New World,Aldous Huxley
+    3,Fahrenheit 451,Ray Bradbury

examples/conversions/xml-interleaved.json

@@ -0,0 +1,19 @@
+{
+  "root": [
+    {
+      "a": "first"
+    },
+    {
+      "b": "second"
+    },
+    {
+      "a": "third"
+    },
+    {
+      "c": "fourth"
+    },
+    {
+      "b": "fifth"
+    }
+  ]
+}

examples/conversions/xml-interleaved.toon

@@ -0,0 +1,6 @@
+root:
+  - a: first
+  - b: second
+  - a: third
+  - c: fourth
+  - b: fifth

examples/conversions/xml-rss.json

@@ -0,0 +1,19 @@
+{
+  "rss": {
+    "version": "2.0",
+    "channel": {
+      "title": "Example Feed",
+      "link": "http://example.com",
+      "item": [
+        {
+          "title": "First Post",
+          "link": "http://example.com/1"
+        },
+        {
+          "title": "Second Post",
+          "link": "http://example.com/2"
+        }
+      ]
+    }
+  }
+}

examples/conversions/xml-rss.toon

@@ -0,0 +1,8 @@
+rss:
+  version: 2.0
+  channel:
+    title: Example Feed
+    link: "http://example.com"
+    item[2]{title,link}:
+      First Post,"http://example.com/1"
+      Second Post,"http://example.com/2"

examples/invalid/xml-undefined-namespace.toon

@@ -0,0 +1,2 @@
+root:
+  ex:child: value

examples/valid/xml-attributes.toon

@@ -0,0 +1,10 @@
+element:
+  id: 123
+  class: container primary
+  "data-value": "quoted: value"
+
+a:
+  - xmlns:xlink: "http://www.w3.org/1999/xlink"
+  - xlink:href: "http://example.com"
+  - xlink:title: Example Link
+  - Click here

examples/valid/xml-mixed-content.toon

@@ -0,0 +1,14 @@
+article:
+  p[2]:
+    - [5]:
+        - "This is "
+        - em: emphasized
+        - " and "
+        - strong: bold
+        - " text."
+    - [3]:
+        - "Another "
+        - a:
+            - href: "http://example.com"
+            - link
+        - " here."

examples/valid/xml-namespaces.toon

@@ -0,0 +1,9 @@
+soap:Envelope:
+  xmlns:soap: "http://schemas.xmlsoap.org/soap/envelope/"
+  xmlns:m: "http://example.com/messages"
+  soap:Header:
+    m:Auth:
+      token: abc123
+  soap:Body:
+    m:GetUser:
+      m:id: 42

examples/valid/xml-repeated-elements.toon

@@ -0,0 +1,5 @@
+root:
+  - a: first
+  - b: second
+  - a: third
+  - b: fourth

examples/valid/xml-xhtml.toon

@@ -0,0 +1,16 @@
+html:
+  xmlns: "http://www.w3.org/1999/xhtml"
+  lang: en
+  head:
+    title: Page Title
+    meta:
+      charset: UTF-8
+  body:
+    class: main
+    h1: Hello World
+    p[3]:
+      - "Welcome to "
+      - a:
+          - href: /
+          - our site
+      - .

package.json

@@ -1,7 +1,7 @@
 {
   "name": "@toon-format/spec",
   "type": "module",
-  "version": "3.0.1",
+  "version": "4.0.0",
   "packageManager": "pnpm@10.19.0",
   "description": "Official specification for Token-Oriented Object Notation (TOON)",
   "author": "Johann Schopplich <hello@johannschopplich.com>",

tests/README.md

@@ -27,7 +27,8 @@ tests/
 │   │   ├── delimiters.json
 │   │   ├── whitespace.json
 │   │   ├── options.json
-│   │   └── key-folding.json
+│   │   ├── key-folding.json
+│   │   └── xml-constructs.json
 │   └── decode/             # Decoding tests (TOON → JSON)
 │       ├── primitives.json
 │       ├── numbers.json
@@ -41,7 +42,8 @@ tests/
 │       ├── validation-errors.json
 │       ├── indentation-errors.json
 │       ├── blank-lines.json
-│       └── path-expansion.json
+│       ├── path-expansion.json
+│       └── xml-constructs.json
 └── README.md               # This file
 ```
 
@@ -160,6 +162,7 @@ The fixture format is language-agnostic JSON, so you can load and iterate it usi
 | `delimiters.json` | Tab and pipe delimiter options | §11 |
 | `whitespace.json` | Formatting invariants and indentation | §12 |
 | `key-folding.json` | Key folding with safe mode, depth control, collision avoidance | §13.4 |
+| `xml-constructs.json` | XML constructs: namespaces, attributes, repeated elements, mixed content | §22-28 |
 
 ### Decoding Tests (`fixtures/decode/`)
 
@@ -178,6 +181,7 @@ The fixture format is language-agnostic JSON, so you can load and iterate it usi
 | `indentation-errors.json` | Strict mode indentation validation | §14.3, §12 |
 | `blank-lines.json` | Blank line handling in arrays | §14.4, §12 |
 | `path-expansion.json` | Path expansion with safe mode, deep merge, strict-mode conflicts | §13.4, §14.5 |
+| `xml-constructs.json` | XML constructs: namespaces, attributes, repeated elements, mixed content, undefined prefix errors | §22-28, §14.6 |
 
 ## Validating Fixtures