Bug: Crash after queuing 30 tasks for the safety worker

[PiotrKorkus]

Original report: https://github.com/qorix-group/inc_orchestrator_internal/issues/395

Describe the bug:

If there are more than 30 error tasks that could not be handled immediately by the safety worker, then crash happens.

1. Spawn 30+ safe tasks, return fail for all of them
2. Prevent immediate handling of the failure e.g. by waiting for all of them in join_all
3. Crash will happen when 31st task returns error.

Steps to reproduce the behavior:

1. git checkout igorostrowskiq_safety_worker_tests
2. cd component_integration_tests/rust_test_scenarios
3. cargo run -- --name runtime.worker.safety_worker.queue_safety_worker_tasks --input '{"runtime": {"task_queue_size": 256, "workers": 1, "safety_worker": {}}, "test": {"task_count": 31}}'

Observed behavior:

Segmentation fault happens

Expected behavior

No segmentation fault. If there is hard limit for such tasks, there should be a proper documentation + handling of such case (panic with detailed info?)

Occurrence:

Always

Attachments / Logs:

crash_log.txt

[prabakaranklst]

Refer comments in https://github.com/qorix-group/inc_orchestrator_internal/issues/395