Airflow 3.0.6 - apache-airflow-providers-celery 3.12.2 - Process 'ForkPoolWorker-' exited with 'signal 9 (SIGKILL)

[hss-rmatthews]

Hi All,

We are running Airflow version 3.0.6 in MWAA and have a task failing without any logs showing in Airflow.

The task log shows:
An error occurred (ResourceNotFoundException) when calling the GetLogEvents operation: The specified log stream does not exist.

However, when investigating the worker logs we are finding the following entries:

[error] Timed out waiting for UP message from <ForkProcess(ForkPoolWorker-245, started daemon)> [celery.concurrency.asynpool]
[ERROR/MainProcess] Process 'ForkPoolWorker-245' pid: exited with 'signal 9 (SIGKILL)'

This particular task is using the S3ToRedshiftOperator and I have verified that the command is actually present and in our Redshift logs and it has succeeded.

The spec of the environment in MWAA is: mw1.medium with a maximum worker count of 20. We currently only have a single DAG running once a day (as we are in the process of migrating from version 2.10.3).

Thanks

Rob

[budaesandrei]

I'm having this too and struggling to find the proper fix. I've tried reducing my celery.worker_autoscale from "20,0" -> "10,0"...I tried adding core.execute_tasks_new_python_interpreter = true. Supposedly that tells celery to start new python interpreters instead of using os.fork() which seems to be the actual issue in python3.12 but that didn't work either. I'm running out of ideas what to try. Even my agent is just saying "you have no option but to wait for AWS to upgrade to 3.1+ with the fix" which is not an option for us using MWAA. And we can't downgrade because we'd lose data.

[potiuk]

This is likely signalling out-of memory issue, not necessarily fork issue., Fork SIGKILL might happen (known issue) on MacOS, but on Linux, forking is rarely a reason for SIGKILL. Usuallly system will SIGKILL you when your start using too much memory