Airflow 3x

[project-defiant]

Context

This PR updated airflow to 3.x.

The deployment of 3.x is different to 2.x, as architecture now includes additional services

• airflow-scheduler - The scheduler monitors all tasks and dags, then triggers the task instances once their dependencies are complete.
• airflow-dag-processor - The DAG processor parses DAG files.
• airflow-api-server - The api server is available at http://localhost:8080.
• airflow-worker - The worker that executes the tasks given by the scheduler.
• airflow-triggerer - The triggerer runs an event loop for deferrable tasks.
• airflow-init - The initialization service.
• postgres - The database.
• redis - The redis - broker that forwards messages from scheduler to worker.

The update includes:

• swapping the airflow-webserver to airflow-apiserver that is only one service that seeks the postgres workloads.
• introducing celery worker and redis, dag-processor (following the official celery setup)

[Copilot]

Pull Request Overview

This PR upgrades Apache Airflow from version 2.x to 3.x, introducing a new microservices-based architecture that replaces the monolithic webserver with separate specialized services for improved scalability and separation of concerns.

• Migrates from LocalExecutor to CeleryExecutor with Redis as message broker
• Replaces airflow-webserver with airflow-apiserver and introduces dedicated services (dag-processor, worker, triggerer)
• Updates import paths and configurations to align with Airflow 3.x API changes

Reviewed Changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/orchestration/utils/labels.py	Updates import path for Context from airflow.utils to airflow.sdk.definitions
src/orchestration/dags/unified_pipeline.py	Updates import paths for task_group and EmptyOperator, removes deprecated default_view parameter
pyproject.toml	Upgrades Airflow to 3.0.6, consolidates dependencies, removes obsolete configurations
compose.yaml	Implements new microservices architecture with separate containers for scheduler, worker, dag-processor, triggerer, and apiserver
compose.local.yaml	Updates service names and adds Google Cloud credentials configuration for new services
Makefile	Adds --build flag to ensure Docker images are rebuilt during development
Dockerfile	Switches to Ubuntu base image, updates user ID, and modernizes build process with uv sync

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

The user ID change from 1000 to 1010 should be documented or explained. This change could cause permission issues in existing deployments where volumes were created with the previous UID.

Suggested change

	RUN useradd --create-home --uid 1010 airflow
	WORKDIR /home/airflow

[Copilot]

The command format is incorrect. It should be 'celery' and 'worker' as separate arguments in the list, not as a single string 'celery worker'.

Suggested change

	- celery worker
	- celery
	- worker