streamdiffusion sdxl - Error in monitor loop (stuck worker)

[eliteprox]

Describe the bug

I found an livepeer/ai-runner:live-app-streamdiffusion-sdxl container that was running for several hours and printing the following logs:

Traceback (most recent call last):
  File "/app/app/live/process/process_guardian.py", line 271, in _monitor_loop
    last_error = self.process.get_last_error()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/app/live/process/process.py", line 449, in get_last_error
    last_error = self.error_queue.get_nowait()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/miniconda3/envs/comfystream/lib/python3.11/multiprocessing/queues.py", line 135, in get_nowait
    return self.get(False)
           ^^^^^^^^^^^^^^^
  File "/workspace/miniconda3/envs/comfystream/lib/python3.11/multiprocessing/queues.py", line 100, in get
    raise ValueError(f"Queue {self!r} is closed")
ValueError: Queue <multiprocessing.queues.Queue object at 0x75bb4b35ca10> is closed
Stack (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/app/app/live/infer.py", line 227, in <module>
    asyncio.run(
  File "/workspace/miniconda3/envs/comfystream/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
  File "/workspace/miniconda3/envs/comfystream/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/workspace/miniconda3/envs/comfystream/lib/python3.11/asyncio/base_events.py", line 641, in run_until_complete
    self.run_forever()
  File "/workspace/miniconda3/envs/comfystream/lib/python3.11/asyncio/base_events.py", line 608, in run_forever
    self._run_once()
  File "/workspace/miniconda3/envs/comfystream/lib/python3.11/asyncio/base_events.py", line 1936, in _run_once
    handle._run()
  File "/workspace/miniconda3/envs/comfystream/lib/python3.11/asyncio/events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
  File "/app/app/live/process/process_guardian.py", line 340, in _monitor_loop
    logging.exception("Error in monitor loop", stack_info=True)
timestamp=2025-11-02 04:23:35 level=ERROR location=process_guardian.py:340:_monitor_loop gateway_request_id=43f4f6e9 manifest_id=ef783006 stream_id=aiJobTesterStream-1762013066117295814 message=Error in monitor loop

Reproduction steps

Running go-livepeer v0.8.8 with livepeer/ai-runner:live-app-streamdiffusion-sdxl on dual 4090 system.

Docker inspect:
"Image": "sha256:f51676ce8332dbad414b9e3daa66a5f5a797e6d54682601d8d910c151a5e1748",
"Image": "livepeer/ai-runner:live-app-streamdiffusion-sdxl",

Commit: a201f99

Expected behaviour

_monitor_loop should not fail and /health endpoint should have reported the error state to allow container restart

Severity

None

Screenshots / Live demo link

No response

OS

None

Running on

None

AI-worker version

No response

Additional context

No response

[eliteprox]

This issue occurred again on another system today. Has been seen on 3 different systems, so not an isolated issue.

I have recently updated to livepeer/ai-runner:live-app-streamdiffusion-sdxl-faceid-sha-a096fa1 and deleted/recompiled engines from the same dl_checkpoints.sh version. This apparently did not resolve the issue as it has happened again.

This time I captured output from ps aux | grep 'python' while the CPU was stuck at 100%. I captured this about 5 minutes after the endless loop began

elite@eliteprd2:~$ ps aux | grep 'python'
root        1183  0.0  0.0 109696  5888 ?        Ssl  Nov18   0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
root        1201  0.1  0.0 261624  8192 ?        Ssl  Nov18   7:04 /usr/bin/python3 -Es /usr/sbin/tuned -l -P
root        2708  0.1  0.0  22176  7168 ?        Ss   Nov18   7:09 python3 /home/elite/nvidia-fan-controller/nvidia_fan_controller4.py --target-temperature 45 --interval-secs 2 --fan-map 0:0,1;1:2,3
root     2037217  0.2  0.0 12262460 15052 ?      Dsl  06:17   1:30 /workspace/miniconda3/envs/comfystream/bin/python3.11 /workspace/miniconda3/envs/comfystream/bin/uvicorn app.main:app --log-config app/cfg/uvicorn_logging_config.json --host  --port 8000
root     2037509 11.7  2.6 19053628 1749956 ?    Sl   06:17  77:05 /workspace/miniconda3/envs/comfystream/bin/python3.11 -u -m app.live.infer --pipeline streamdiffusion-sdxl-faceid --http-port 8888 --initial-params {  "model_id": "stabilityai/sdxl-turbo",  "prompt": "mona lisa, face-focused webcam portrait blooming into stained-glass prisms and aurora ribbons, retro studio glow, friendly cardigan vibes, dreamlike, cinematic, ultra-detailed",  "prompt_interpolation_method": "slerp",  "normalize_prompt_weights": true,  "negative_prompt": "blurry, low quality, flat, 2d",  "guidance_scale": 1,  "delta": 0.7,  "num_inference_steps": 50,  "t_index_list": [    36  ],  "width": 512,  "height": 512,  "lora_dict": null,  "use_lcm_lora": true,  "lcm_lora_id": "latent-consistency/lcm-lora-sdv1-5",  "acceleration": "tensorrt",  "use_denoising_batch": true,  "do_add_noise": true,  "seed": 789,  "seed_interpolation_method": "linear",  "normalize_seed_weights": true,  "enable_similar_image_filter": false,  "similar_image_filter_threshold": 0.98,  "similar_image_filter_max_skip_frame": 10,  "controlnets": [    {      "model_id": "xinsir/controlnet-depth-sdxl-1.0",      "conditioning_scale": 0.4,      "preprocessor": "depth_tensorrt",      "preprocessor_params": {},      "enabled": true,      "control_guidance_start": 0,      "control_guidance_end": 1    },    {      "model_id": "xinsir/controlnet-canny-sdxl-1.0",      "conditioning_scale": 0.1,      "preprocessor": "canny",      "preprocessor_params": {},      "enabled": true,      "control_guidance_start": 0,      "control_guidance_end": 1    },    {      "model_id": "xinsir/controlnet-tile-sdxl-1.0",      "conditioning_scale": 0.1,      "preprocessor": "feedback",      "preprocessor_params": {},      "enabled": true,      "control_guidance_start": 0,      "control_guidance_end": 1    }  ],  "ip_adapter": {    "type": "faceid",    "scale": 1,    "enabled": true  },  "ip_adapter_style_image_url": "https://storage.googleapis.com/lp-ai-assets/ipadapter_style_imgs/faceid/mona_lisa.jpeg"}
root     2037551  0.0  0.0  15636  2304 ?        S    06:17   0:00 /workspace/miniconda3/envs/comfystream/bin/python3.11 -c from multiprocessing.resource_tracker import main;main(8)
root     2037553 43.3 43.6 129485192 28682680 ?  Dl   06:17 285:15 /workspace/miniconda3/envs/comfystream/bin/python3.11 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=9, pipe_handle=21) --multiprocessing-fork
root     2063641  0.2  0.0 12262444 13524 ?      Dsl  07:16   1:22 /workspace/miniconda3/envs/comfystream/bin/python3.11 /workspace/miniconda3/envs/comfystream/bin/uvicorn app.main:app --log-config app/cfg/uvicorn_logging_config.json --host  --port 8000
root     2063891 15.0  2.0 17283252 1377008 ?    Dl   07:16  90:01 /workspace/miniconda3/envs/comfystream/bin/python3.11 -u -m app.live.infer --pipeline streamdiffusion-sdxl-faceid --http-port 8888 --initial-params {  "model_id": "stabilityai/sdxl-turbo",  "prompt": "mona lisa, face-focused webcam portrait blooming into stained-glass prisms and aurora ribbons, retro studio glow, friendly cardigan vibes, dreamlike, cinematic, ultra-detailed",  "prompt_interpolation_method": "slerp",  "normalize_prompt_weights": true,  "negative_prompt": "blurry, low quality, flat, 2d",  "guidance_scale": 1,  "delta": 0.7,  "num_inference_steps": 50,  "t_index_list": [    36  ],  "width": 512,  "height": 512,  "lora_dict": null,  "use_lcm_lora": true,  "lcm_lora_id": "latent-consistency/lcm-lora-sdv1-5",  "acceleration": "tensorrt",  "use_denoising_batch": true,  "do_add_noise": true,  "seed": 789,  "seed_interpolation_method": "linear",  "normalize_seed_weights": true,  "enable_similar_image_filter": false,  "similar_image_filter_threshold": 0.98,  "similar_image_filter_max_skip_frame": 10,  "controlnets": [    {      "model_id": "xinsir/controlnet-depth-sdxl-1.0",      "conditioning_scale": 0.4,      "preprocessor": "depth_tensorrt",      "preprocessor_params": {},      "enabled": true,      "control_guidance_start": 0,      "control_guidance_end": 1    },    {      "model_id": "xinsir/controlnet-canny-sdxl-1.0",      "conditioning_scale": 0.1,      "preprocessor": "canny",      "preprocessor_params": {},      "enabled": true,      "control_guidance_start": 0,      "control_guidance_end": 1    },    {      "model_id": "xinsir/controlnet-tile-sdxl-1.0",      "conditioning_scale": 0.1,      "preprocessor": "feedback",      "preprocessor_params": {},      "enabled": true,      "control_guidance_start": 0,      "control_guidance_end": 1    }  ],  "ip_adapter": {    "type": "faceid",    "scale": 1,    "enabled": true  },  "ip_adapter_style_image_url": "https://storage.googleapis.com/lp-ai-assets/ipadapter_style_imgs/faceid/mona_lisa.jpeg"}
root     2063932  0.0  0.0  15636  3840 ?        S    07:16   0:00 /workspace/miniconda3/envs/comfystream/bin/python3.11 -c from multiprocessing.resource_tracker import main;main(8)
root     2063934 55.3 46.9 154538632 30843436 ?  Dl   07:16 331:07 /workspace/miniconda3/envs/comfystream/bin/python3.11 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=9, pipe_handle=21) --multiprocessing-fork
root     2301423 89.3  0.0 112780 38912 ?        R    17:14   0:14 /usr/bin/python3 /usr/lib/ubuntu-release-upgrader/check-new-release -q