feat: add initial extension support #395

wackywendell · 2025-09-15T19:42:24Z

Summary

Begins addressing #367 and #342 by adding support for parsing the types in YAML Simple Extension Files into Rustic types - with validity enforced. This includes a string text parser handling built-in types, compound types, named structs, custom types, and validated parameter constraints in the Simple Extension YAML files.

Scope

Types-only: no functions or call validation yet.
Public API exposes parsed types (ExtensionFile, Registry, CustomType, ConcreteType) and enforces validation of those on creation / read.

Key Changes

Type system
- New BuiltinType, CompoundType, ConcreteType, CustomType with Display/round‑trip support for alias and named‑struct structures.
- Parameter constraints: data type, integer (with min/max), enum (validated/deduped), boolean, string.
- Parsing to and from the YAML structures (TryFrom<TypeParamDefsItem>, Parse<RawType>)
File/Registry type: abstraction for handling YAML files
Context and proto glue
- Separates out ProtoContext from Context, to distinguish between things needed for Protobuf parsing (ProtoContext)
Type expression parser
- Parses simple, user‑defined (u!Name) and type variables; visits extension references for linkage bookkeeping.
Build/CI
- parse feature includes serde_yaml; include!(extensions.in) is gated behind extensions feature.
- Aligns actions/checkout to v4, updates Cargo dependency set, and bumps the substrait submodule.

Compatibility Notes

New trait bound ProtoContext on proto parsing that previously required only Context.
extensions.in now compiled only with features=["extensions"].
Minimal, types-only round‑trip implemented; other sections remain empty when converting back to text.

Testing

New unit tests cover:
- Type parsing and round‑trip for alias and named‑struct.
- Parameter constraint handling including enum validation and integer bounds (with current truncation behavior).
- Registry creation and type lookup; core registry smoke test behind features=["extensions"].

…ension types

benbellick

Left a few comments but still haven't gone through most of the PR. Just wanted to flush what I have so far, but I will review more later. Thanks!

benbellick · 2025-12-01T22:57:18Z

Cargo.toml

 prost = "0.14.1"
 prost-types = "0.14.1"
+# Required by generated text schemas: the typify-generated code emits
+# ::regress::Regex for `pattern` validations.


Just to clarify, it looks like regress was already present before. Is this comment just clarifying why it was there in the first place?

Yes, just clarifying its use!

benbellick · 2025-12-01T22:58:54Z

src/parse/context.rs

+pub trait ProtoContext: Context {
    /// Add a [SimpleExtensionUrn] to this context. Must return an error for duplicate
-    /// anchors or when the urn is not supported.
+    /// anchors or when the URI is not supported.


Can we just consistently use URN here? We have support for both in all of the other libraries and we will be dropping URI eventually.

benbellick · 2025-12-02T00:39:27Z

src/parse/context.rs

    }
+}

+pub trait ProtoContext: Context {


As mentioned, I don't have a ton of Rust experience, so it may be that this is totally standard.

But I wonder if there's a simpler way to handle parsing that isn't so trait-heavy. AFAICT, there is only one implementor of the ProtoContext trait, which is the test fixture (lines 81-107).

Could we drop the ProtoContext trait and use a concrete type instead, then update tests to use explicit instantiations of that type? If not, what is the benefit of using this trait here?

benbellick

Left a few more comments, but excited to get this in! Sorry that this PR has been sitting so long 😅

benbellick · 2025-12-02T15:08:11Z

src/parse/text/simple_extensions/registry.rs

+impl Registry {
+    /// Create a new Global Registry from validated extension files
+    pub fn new(extensions: Vec<ExtensionFile>) -> Self {
+        Self { extensions }


What happens if two extension files contain conflicting definitions? E.g. two different files have the same urn but introduce types with the same name but different parameters.

I wonder if it makes sense to instead maintain a hashmap from URN to ExtensionFile? The spec says:

Each YAML file must include a required urn field that uniquely identifies the extension.

To me, this suggests that (at least for now) all URNs must uniquely identify a single extension file.

benbellick · 2025-12-02T15:09:07Z

src/parse/text/simple_extensions/registry.rs

+        Self { extensions }
+    }
+
+    // Private helper methods


This comment suggests that the below methods are private but get_type below is public.

Suggested change

// Private helper methods

benbellick · 2025-12-02T16:05:50Z

src/parse/text/simple_extensions/file.rs

+        }
+
+        let back = ext.to_raw();
+        assert_eq!(back.urn, "extension:example.com:param_test");


Can we simplify this test by just generating the PartialEq implementation for text::simple_extensions::SimpleExtensions and then just directly checking roundtrip equality here (rather than digging into individual fields). We will probably want to introduce a roundtrip test framework at some point to be consistent with the other implementations, so might as well start here 🙂

benbellick · 2025-12-02T16:11:31Z

src/parse/text/simple_extensions/parsed_type.rs

+
+impl<'a> TypeExpr<'a> {
+    /// Parse a type string into a [`TypeExpr`].
+    pub fn parse(type_str: &'a str) -> Result<Self, TypeParseError> {


I don't have a particular suggestion here, but just wanted to call it that it would be awesome if we could use the antlr grammar here. There aren't great rust bindings for antlr (there is this but it was last updated 3 years ago). That being said, maybe an evil solution in the future is to generate C++ bindings and then call those from rust 🤷

benbellick · 2025-12-02T16:13:28Z

src/parse/text/simple_extensions/parsed_type.rs

+
+    #[test]
+    fn test_user_defined_and_parameters() {
+        let expr = "u!geo?<i32?, point<i32, i32>>";


Is it possible to follow the structure above of just having a list of strings and expected types that we iterate over? I can imagine wanting to add a few more cases in the future.

To be clear, I mean something like:

let test_cases = [ (case1, expect1), (case2, expect2), etc.. ];

wackywendell and others added 15 commits August 12, 2025 16:38

feat: ExtensionRegistry, wip

076998e

wip compiles

57e5dee

working through some implications

68f0e7d

Switched types over, compiles

dfef7ee

Stripped down to just types

facb09b

Enable test only with correct feature flag

5a5ac23

Code cleanup

7869676

Merged with existing parse

bc2a86b

Mostly working but builtin types are a bit messed up

1b49a6c

now looks better

6dc4302

On the way, trying to get validation working

9c593a1

Extensions should be included when the extensions feature is enabled

7a8584e

Delete some dead code

ac447d8

Start merging argument and types

9b47219

Some updates to type handling; compiles and passes tests

8a44db4

wackywendell marked this pull request as draft September 15, 2025 20:20

wackywendell mentioned this pull request Sep 15, 2025

Feature: Add a 'Rustic' Ergonomic and Validating Layer #367

Open

wackywendell added 13 commits September 15, 2025 17:20

Merge remote-tracking branch 'upstream/main' into parse-extensions

47fbf18

Update to match URN change

5b1e5df

Some renames

6444f5c

Update tests

162e8ab

Removed string / enum type parameter, that's not a thing

421d957

Time should be primitive, precisiontime parameterized

46b6468

Enforce ranges on fixed-range types

83a4249

Enforce float / int bounds

f415dc8

Validate missing types

0cbfc34

Add error for type variations

93b41ef

Fix casing on type display, add tests

470aa00

Stable field ordering for structures

202747a

fix(parse): apply Substrait identifier rules for CustomType names

ac479d9

docs(parse): add SPDX header, clarify errors, and guard duplicate ext…

c0a60b4

…ension types

wackywendell changed the title ~~Add Initial Extension Support~~ feat: Add Initial Extension Support Oct 10, 2025

wackywendell changed the title ~~feat: Add Initial Extension Support~~ feat: add initial extension support Oct 10, 2025

wackywendell added 2 commits October 10, 2025 11:23

fix: clippy warning about nested ifs

ca1e133

doc: update extensions types and links

95270ab

wackywendell force-pushed the parse-extensions branch from 07c6293 to 95270ab Compare October 10, 2025 12:11

wackywendell added 2 commits October 10, 2025 16:19

Merge remote-tracking branch 'upstream/main' into parse-extensions

25d6392

fix: update usage of typify-generated types to match typify upgrade

3e58a42

wackywendell force-pushed the parse-extensions branch from 01172ec to 3e58a42 Compare October 10, 2025 14:25

wackywendell marked this pull request as ready for review October 20, 2025 15:32

benbellick reviewed Dec 2, 2025

View reviewed changes

feat: add initial extension support #395

Are you sure you want to change the base?

feat: add initial extension support #395

Conversation

wackywendell commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Scope

Key Changes

Compatibility Notes

Testing

Uh oh!

benbellick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benbellick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wackywendell commented Sep 15, 2025 •

edited

Loading