mlflow integration

[samhita-alla]

Summary

• New flyteplugins-mlflow plugin that integrates MLflow experiment tracking with Flyte tasks via a @mlflow_run decorator
• Supports automatic run management, autologging, parent/child run sharing, nested runs (cross-container), distributed training (rank-gated logging), and auto-generated MLflow UI links
• Includes configuration management via mlflow_config() for workflow-level defaults and per-task overrides

Run modes

• auto (default): Reuse parent's MLflow run if available, otherwise create a new one
• new: Always create an independent run
• nested: Create a new run with mlflow.parentRunId tag linking it to the parent. This works across tasks without needing the parent run active in the same process.

Key features

• Autologging: @mlflow_run(autolog=True, framework="sklearn") enables MLflow autologging with optional framework targeting
• Distributed training: Auto-detects RANK env var; only rank 0 logs to MLflow
• Link auto-generation: mlflow_config(link_host=...) generates MLflow UI URLs that propagate to child tasks with links=(Mlflow(),)
• Local execution: Handles MLflow's active run stack correctly when parent and child tasks share a process

[cosmicBboy]

@samhita-alla let's add the mlflow plugin to the publish.yaml CI https://github.com/flyteorg/flyte-sdk/blob/main/.github/workflows/publish.yml

[samhita-alla]

@cosmicBboy i keep forgetting. sorry about that. it should be added now.