compat-gen: Epoch A adapter (v0.36.0)#6892
Draft
joseph-isaacs wants to merge 6 commits intovortex-data:developfrom
Draft
compat-gen: Epoch A adapter (v0.36.0)#6892joseph-isaacs wants to merge 6 commits intovortex-data:developfrom
joseph-isaacs wants to merge 6 commits intovortex-data:developfrom
Conversation
Standalone crate for generating backward-compat fixture .vortex files. Uses Epoch A adapter targeting the v0.36.0 write API: - VortexWriteOptions::default() (no session) - .write(sink, stream).await returns the sink Fixtures: primitives, strings, booleans, nullable, struct_nested, chunked, tpch_lineitem, tpch_orders. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Write API is the same as Epoch A but with stricter stream bounds (Send + 'static). Also has write_blocking() for sync usage. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Session-based API: VortexSession::default() + session.write_options(). write() now returns WriteSummary and takes &mut sink. This is the adapter that compiles against the current codebase. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds: compat-test binary design, ClickBench fixture, per-encoding stubs, validation strategy (ChunkedArray + assert_arrays_eq!), reqwest-based HTTPS fetching, CI workflow specs, and complete code size / shared-vs-branch-specific breakdown. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…workflows - adapter.rs: add read_file() for Epoch C (session-based open_buffer + async scan) - fixtures/clickbench.rs: download pinned ClickBench hits_0.parquet, take 1k rows - fixtures/encodings.rs: todo!() stubs for Dict, RunEnd, Constant, Sparse, ALP, BitPacked, FSST - test_main.rs: CLI with --fixtures-url (HTTPS) or --fixtures-dir (local) - validate.rs: fetch manifest + fixtures, compare via ChunkedArray + assert_arrays_eq! - compat-gen-upload.yml: generate + upload to S3 on tag push, updates versions.json - compat-test-weekly.yml: weekly validation of all versions against HEAD reader - Cargo.toml: add reqwest, parquet deps + compat-test binary target Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
VortexWriteOptions::default(), async in-memory open, into_array_stream(). Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
vortex-test/compat-gen/crate for generating backward-compat fixture.vortexfilesVortexWriteOptions::default().write(sink, stream)This PR is part of a 3-PR series showing how
adapter.rschanges per API epoch. See RFC 23.Crate layout
Test plan
cargo build --manifest-path vortex-test/compat-gen/Cargo.tomlcompilescargo run --manifest-path vortex-test/compat-gen/Cargo.toml --bin compat-gen -- --version 0.36.0 --output /tmp/fixtures/produces fixture files🤖 Generated with Claude Code