Skip to content

Add IBM MQ provider#62790

Draft
dabla wants to merge 10 commits intoapache:mainfrom
dabla:feature/ibm-mq-provider
Draft

Add IBM MQ provider#62790
dabla wants to merge 10 commits intoapache:mainfrom
dabla:feature/ibm-mq-provider

Conversation

@dabla
Copy link
Contributor

@dabla dabla commented Mar 3, 2026

With the introduction of event-driven scheduling and the MessageQueueProvider abstraction in Airflow, it has become significantly easier to trigger DAGs from external message brokers as described in Astronomer's guide on event-driven scheduling.

Many enterprises still rely heavily on IBM MQ as their primary enterprise messaging backbone. However, at the moment there is no official Airflow provider supporting IBM MQ.

This implementation consists of:

  • An IBMMQHook
  • A MessageQueueProvider implementation for IBM MQ
  • The ability to trigger DAGs from IBM MQ events
  • Standard producer/consumer patterns from within tasks

This allows IBM MQ to function similarly to Kafka, SQS, etc., within the Airflow event-driven scheduling framework.

The implementation is built on top of the open-source IBM MQ Python wrapper:

  • IBM MQ Python ibmmq library

IBM has recently released and documented their modern Python binding here:

https://community.ibm.com/community/user/blogs/dylan-goode/2025/10/16/new-python-binding-for-ibm-mq

The hook supports:

  • Secure connections (TLS) (to be done)
  • Queue get/put operations
  • Configurable polling behaviour

The MessageQueueProvider implementation integrates with Airflow's event-driven scheduling so that DAGs can be triggered based on IBM MQ messages.

Why this might make sense:

  • IBM MQ is still widely used in regulated industries (banking, insurance, government).
  • Many enterprises using Airflow also run IBM MQ.
  • This would allow IBM MQ to be a first-class citizen in Airflow's event-driven ecosystem.
  • The dependency is officially maintained by IBM and open source.

I am willing to act as initial maintainer and code owner, of course this is purely a proposition.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@vincbeck
Copy link
Contributor

vincbeck commented Mar 3, 2026

Since this PR needs approval the community, maybe put in draft to avoid any accidental merge?

@dabla dabla marked this pull request as draft March 3, 2026 15:25
@jscheffl
Copy link
Contributor

jscheffl commented Mar 4, 2026

Looks good for me. Would you add yourself as CODEOWNER for this? Do you know somebody (even non committer) which would help as steward?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools area:providers backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch kind:documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants