Metaxy is a metadata layer for multi-modal Data and ML pipelines. Metaxy tracks feature versions, (partial) data dependencies, and lineage across complex computation graphs.
Metaxy's goal is to provide a standard instrument for any kind of multi-modal (or just tabular) incremental pipelines, standardizing dependency specification, versioning, partial data dependencies, manipulations over metadata, migrations, and more.
Read the docs to learn more.
Warning: Metaxy hasn't been publicly released yet, but you can try the latest dev release:
pip install --pre metaxyMetaxy is highly pluggable and generally can be used with any kind of incremental pipelines, storage, metadata stores, and dataframe libraries.
Metaxy provides integrations with popular tools such as Dagster, ClickHouse, DeltaLake, SQLModel.
The full list can be found here.
See CONTRIBUTING.md.
