Unhandled exceptions in worker not visible

[iangoldby]

If there is an unhandled exception in a Worker task, i.e. in Worker.run_task() or Worker.cancel_task(), then it is caught at the end of Worker._handle_task_operation() and the task state is set to failed (which is good of course). But the exception itself is suppressed and there is no logging.

This can make it very difficult to diagnose problems in the agent code run by Worker because it appears that such unhandled exceptions are silently swallowed and the task just mysteriously fails.

I would suggest that the full exception and traceback should be logged using the standard Python logger:

        except Exception:
            logger.exception("Unhandled task exception %s", task_operation)
            await self.storage.update_task(task_operation['params']['id'], state='failed')

In case anyone should counter that this is exactly what the OpenTelemetry integration is for, I would point out that not everyone has OpenTelemetry properly configured in their dev environment, and good console logging is a very simple and fool-proof way of making errors visible that works anywhere. Please could we have this?

Many thanks.