Skip to content

Conversation

@HJLebbink
Copy link
Member

Description

Complete implementation of AWS S3 Tables / Apache Iceberg catalog support for MinIO Rust SDK. Provides type-safe interface for managing warehouses,
namespaces, and tables with full HTTP execution, authentication, and error handling across 20 operations.

Motivation and Context

Enables Rust applications to interact with MinIO AIStor's Iceberg catalog through S3 Tables API. Implements AWS-compatible REST catalog with ACID
transaction guarantees for table metadata management.

How to test this PR?

# Run unit tests
cargo test --test tables_unit_tests

# Run quickstart example (requires MinIO AIStor instance)
cargo run --example tables_quickstart --features localhost

# Build library
cargo build --features localhost

Types of changes

- New feature (non-breaking change which adds functionality)

Checklist:

- Unit tests added/updated (16 passing tests)
- Internal documentation updated
- Create a documentation update request https://github.com/miniohq/aistor-object-store-docs/issues/new

---
Implementation Summary

Core Infrastructure (Phases 1-8)

- 20 Operations: Full warehouse, namespace, and table lifecycle management
- Type-Safe Builders: Compile-time validation using typed_builder pattern
- HTTP Execution: Custom execute_tables() with path-based routing
- Authentication: S3 Signature V4 with s3tables service name
- Error Handling: 15+ typed error variants with server error mapping

Operations Implemented

Warehouse Operations (4)
- CreateWarehouse, ListWarehouses, GetWarehouse, DeleteWarehouse

Namespace Operations (4)
- CreateNamespace, ListNamespaces, GetNamespace, DeleteNamespace

Table Operations (7)
- CreateTable, RegisterTable, LoadTable, ListTables, DeleteTable, RenameTable, CommitTable

Advanced Operations (3)
- CommitMultiTableTransaction, GetConfig, TableMetrics

Iceberg Type System

- Complete schema definitions (Schema, Field, FieldType)
- Partition specifications and transforms
- Sort orders and snapshot management
- Table metadata and commit operations

Key Features

- Multi-level namespace support with Unit Separator encoding
- Pagination support for list operations
- Optimistic concurrency control for commits
- Storage credentials for data access
- JSON serialization/deserialization throughout

Files Modified/Created

- 23 files modified: Core infrastructure (client, signer, error handling)
- 69 files created: Builders, responses, types (~8,500 lines)
- 3 documentation files: Implementation plan, architecture, HTTP guide
- 2 test files: Unit tests (16 tests), quickstart example

Code Statistics

- Lines Added: ~8,750 lines total
  - Implementation: ~6,500 lines
  - Documentation: ~2,000 lines
  - Tests: ~250 lines
- Test Coverage: 16 unit tests covering types, builders, and error handling
- Build Status: Clean build with zero warnings

API Usage Example

use minio::s3::{MinioClient, creds::StaticProvider, http::BaseUrl};
use minio::s3::tables::{TablesApi, TablesClient};
use minio::s3::tables::iceberg::{Schema, Field, FieldType, PrimitiveType};

let client = MinioClient::new(base_url, Some(provider), None, None)?;
let tables = TablesClient::new(client);

// Create warehouse
tables.create_warehouse("analytics").build().send().await?;

// Create namespace
tables.create_namespace("analytics", vec!["events".into()])
    .build().send().await?;

// Define schema
let schema = Schema {
    schema_id: 0,
    fields: vec![
        Field {
            id: 1,
            name: "id".into(),
            required: true,
            field_type: FieldType::Primitive(PrimitiveType::Long),
            doc: None,
        },
    ],
    identifier_field_ids: Some(vec![1]),
};

// Create table
tables.create_table("analytics", vec!["events".into()], "clicks", schema)
    .build().send().await?;

Documentation

- TABLES_README.md - User-facing documentation and quick start
- TABLES_IMPLEMENTATION_PLAN.md - Complete 11-phase implementation plan
- TABLES_HTTP_IMPLEMENTATION_GUIDE.md - HTTP execution implementation details
- TABLES_ARCHITECTURE_DECISION.md - Architectural rationale
- examples/tables_quickstart.rs - End-to-end usage example
- tests/tables_unit_tests.rs - Comprehensive unit test suite

@HJLebbink HJLebbink requested a review from Copilot October 21, 2025 19:22
@HJLebbink HJLebbink self-assigned this Oct 21, 2025
@HJLebbink HJLebbink added the enhancement Used in release doc generation label Oct 21, 2025

This comment was marked as outdated.

@HJLebbink HJLebbink force-pushed the s3_tables branch 8 times, most recently from c9bdb5a to 865688d Compare October 23, 2025 13:53
@HJLebbink HJLebbink requested a review from twuebi October 23, 2025 13:57
Copy link
Contributor

@twuebi twuebi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a number of .md files that probably do not belong to this PR.

I'd also use iceberg-rust instead of re-implementing the REST catalog client in minio-rs. I'm not sure what we gain from having those available locally - any real-world use will be using iceberg-rust. If we want to test compatibility, we can also re-export iceberg-rust instead.

implemented everything

added tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Used in release doc generation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants