Skip to content

Add deferrable mode to AzureContainerInstancesOperator#62772

Open
cruseakshay wants to merge 3 commits intoapache:mainfrom
cruseakshay:feature/aci-deferrable-62433
Open

Add deferrable mode to AzureContainerInstancesOperator#62772
cruseakshay wants to merge 3 commits intoapache:mainfrom
cruseakshay:feature/aci-deferrable-62433

Conversation

@cruseakshay
Copy link
Contributor

Add deferrable=True support to AzureContainerInstancesOperator so
tasks release their worker slot while waiting for long-running containers,
offloading polling to the lightweight Triggerer process.

Changes

  • New AzureContainerInstanceAsyncHook — async counterpart to the
    existing sync hook, using azure.mgmt.containerinstance.aio for
    non-blocking state/log/delete calls.

  • New AzureContainerInstanceTrigger — polls ACI at a configurable
    interval and yields a TriggerEvent when the container reaches a
    terminal state.

  • AzureContainerInstancesOperator — three new parameters:

    • deferrable (default from conf) — enables deferrable mode
    • polling_interval (default 5.0s) — trigger poll frequency
    • remove_on_success (default True) — controls cleanup on success
    • Also fixes a bug where the finally cleanup block would delete a
      still-running container group immediately after self.defer() raised
      TaskDeferred.
  • provider.yaml + get_provider_info.py updated to register the
    new trigger.

closes: #62433


Was generative AI tooling used to co-author this PR?

@cruseakshay cruseakshay marked this pull request as ready for review March 3, 2026 10:30
@cruseakshay cruseakshay requested a review from dabla as a code owner March 3, 2026 10:30
Raises an AirflowException on failure; returns the exit code on success.
"""
if event is None:
raise AirflowException("Trigger error: event is None")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for implementing this!

Nit: per the contributing guide, we avoid raising AirflowException directly. Could you raise a built-in exception here instead? https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#dont-raise-airflowexception-directly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add parameter deferrable to AzureContainerInstanceOperator

2 participants