Skip to content

Conversation

@leszko
Copy link
Collaborator

@leszko leszko commented Dec 23, 2025

Based on #241 and #173

This PR extends the plugin support introduced in #241 by running each plugin:

  • in its own uv environment
  • in a separate process

This allows plugins to have conflicting dependencies without interfering with each other.

demo_plusing_separate_environment.mp4

Why this PR?

This PR is primarily a proof of concept to demonstrate that:

  1. Plugins can run in isolated environments without adding significant complexity.
  2. Bringing this to production quality should be straightforward (estimated 1–3 days of work).
  3. Performance is comparable to running everything in a single process.
    • Verified on the long-lived pipeline on Linux: output FPS is unchanged.
  4. The existing plugin interface mostly survives the move from a shared environment to isolated ones. Only the main daydream-scope dependency needs to be changed, see:

How does it work?

  1. During uv daydream-scope install <url>, the plugin code is copied to ~/.daydream-scope/plugins/<plugin-name>
  2. Each plugin includes a pyproject.toml that depends on the base plugin dependencies defined in
    worker_env/pyproject.toml.
  3. When the pipeline is loaded, a dedicated process is started for each plugin using uv run.
  4. Communication between the main process and plugin processes is handled via ZeroMQ.

Side notes

  • This PR also introduces a separate pyproject.toml for each core pipeline, though we may not want to do it
  • The ZeroMQ interface between processes is best understood by reviewing
    https://github.com/daydreamlive/scope/blob/rafal/subprocess-3/src/scope/server/frame_processor_proxy.py
  • If we decide to pursue this approach, there are still several areas to validate, including:
    • Windows support
    • Electron app compatibility
    • Proper process lifecycle and shutdown handling
  • I haven't focused too much on the code, so there is a lot of code that needs refactoring

leszko and others added 30 commits November 24, 2025 08:05
Pipeline proxy was added in PR #173 but is not actually used.
The pipeline is now accessed directly in the worker process through
WorkerFrameProcessor, so the proxy is unnecessary.
Signed-off-by: Yondon Fu <yondon.fu@gmail.com>
Signed-off-by: Yondon Fu <yondon.fu@gmail.com>
Signed-off-by: Yondon Fu <yondon.fu@gmail.com>
Signed-off-by: Yondon Fu <yondon.fu@gmail.com>
Signed-off-by: Yondon Fu <yondon.fu@gmail.com>
Signed-off-by: Yondon Fu <yondon.fu@gmail.com>
Signed-off-by: Yondon Fu <yondon.fu@gmail.com>
…for prompts box

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>
Signed-off-by: Rafał Leszko <rafal@livepeer.org>
Signed-off-by: Yondon Fu <yondon.fu@gmail.com>
Signed-off-by: Yondon Fu <yondon.fu@gmail.com>
Signed-off-by: Yondon Fu <yondon.fu@gmail.com>
…ironments

This merge combines:
- rafal/subprocess-3: Running each pipeline in a separate process with its own environment
- yf/plugins: Plugin support with pluggy-based plugin system

Key changes for plugin isolation:
- Plugins are installed to ~/.daydream-scope/plugins/<plugin_id>/
- Each plugin has its own pyproject.toml with dependencies
- Worker process is started from the plugin's directory with uv run
- Modified install/uninstall CLI commands to copy plugin code to plugins directory
- Worker can dynamically load plugin pipelines from the current working directory

This allows plugins to have their own isolated dependencies that don't conflict
with the main scope environment or other plugins.
@leszko leszko changed the title Rafal/subprocess 3 PoC: Plugins in separate environments Dec 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants