Why do pip packages apache-airflow-providers-* and apache-airflow have cyclic dependencies?

[smktpd]

I use pipdeptree to watch dependency graphs of pip packages in my images.
With image docker.io/apache/airflow:2.11.0-python3.12 it notifies me of such cyclic dependencies:

Warning!!! Cyclic dependencies found:
* apache-airflow-providers-common-compat => apache-airflow => apache-airflow-providers-common-compat
* apache-airflow-providers-common-io => apache-airflow => apache-airflow-providers-common-io
* apache-airflow-providers-common-sql => apache-airflow => apache-airflow-providers-common-sql
* apache-airflow-providers-fab => apache-airflow-providers-common-compat => apache-airflow => apache-airflow-providers-fab
* apache-airflow-providers-ftp => apache-airflow => apache-airflow-providers-ftp
* apache-airflow-providers-http => apache-airflow => apache-airflow-providers-http
* apache-airflow-providers-imap => apache-airflow => apache-airflow-providers-imap
* apache-airflow-providers-smtp => apache-airflow => apache-airflow-providers-smtp
* apache-airflow-providers-sqlite => apache-airflow => apache-airflow-providers-sqlite
* apache-airflow => apache-airflow-providers-common-compat => apache-airflow

I understand that this got to be non-critical, but still feels off, so I wonder why they are made so.

[potiuk]

Because Cyclic dependencies are perfectly OK in Python resolution and this is the only way you can have pre-installed providers.

[smktpd]

Why have them pre-installed? Why not install them?
Wouldn't it be better to avoid circular dependencies and have them listed separately in the dockerfile instructions that are used to produce your images?

[potiuk]

Why do you think everyone uses docker ?

[smktpd]

There are multiple correct answers to this question, but the main reason is isolation via containerization, of course.
Then docker became quite a popular platform with lots of systems already made by someone else (so you don't have to do it yourself).

I don't understand what this question has to do with the topic.
Why keep circular deps? Why not organize dependencies so that there would be no circular deps?

[potiuk]

Becaue your solution will not work outside of docker - 30% of our users do not use any containerisation - so any solutions based on docker will not help those users, so proposing it makes no sense. See https://airflow.apache.org/blog/airflow-survey-2025/

Cyclic dependencies (for now) is needed until we finish task-isolation work, Currently providers depend on apache-airflow but in airflow 3.2 or 3.3 - they are supposed to only depend on a new task-sdk distribution rather than on apache-airflow.

Removing those cyclic dependencies was one of the goals of https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-72+Task+Execution+Interface+aka+Task+SDK which is still not completed.

You can read more about history and reasoning there.

[smktpd]

I was not aware that there already were plans of getting away from cyclic dependencies. The existence of such plans basically answers the initial question.
Thanks!