feat: Add Hugging Face Datasets plugin

[andreahlert]

Summary

Port of the flytekit-huggingface plugin to flyte-sdk v2, enabling native support for datasets.Dataset as a Flyte DataFrame type.

• DataFrameEncoder/Decoder for datasets.Dataset with Parquet serialization
• Cloud storage support (S3, GCS, Azure) via fsspec-compatible storage options
• Anonymous S3 fallback for public datasets (mirrors Polars plugin pattern)
• Column filtering on both encode and decode via type annotations
• Auto-registration via flyte.plugins.types entry point

Follows the same patterns as the existing Polars plugin.

Usage Example

import flyte
import datasets

env = flyte.TaskEnvironment("hf-example")

@env.task
async def create_dataset() -> datasets.Dataset:
    return datasets.Dataset.from_dict({
        "text": ["hello", "world"],
        "label": [0, 1],
    })

@env.task
async def process(ds: datasets.Dataset) -> int:
    return len(ds)

Test plan

• Type recognition tests (Dataset, with columns, with format annotations)
• Handler registration and property tests
• Encode/decode roundtrip tests
• DataFrame wrapper integration test
• Raw task I/O test via flyte.TaskEnvironment
• Column subsetting on decode
• Various data types (int, float, str, bool)
• Empty dataset roundtrip
• Storage options tests (S3, GCS, Azure, anonymous, unknown protocol)

[andreahlert]

@pingsutw @cosmicBboy @kumare3 could you take a look? This ports the huggingface plugin from flytekit to v2, same approach as the polars plugin.

[andreahlert]

Hi Ketan! With 2.0 out I’ve rebased and addressed several of your comments: using the public storage API instead of get_storage, and public flyte.io imports where possible.

[andreahlert]

Addressed all comments. Switched both encode and decode to use pq.write_table/pq.read_table with the Flyte filesystem directly. This avoids HuggingFace's DownloadManager cache entirely and keeps credentials handled by Flyte's storage layer.

As a side effect, _get_storage_options is gone and the empty dataset test is no longer skipped since that was a from_parquet limitation. Also dropped column filtering from encode to match the Polars plugin pattern.