Add initial support for plugins #241

yondonfu · 2025-12-17T02:35:03Z

Demo

2025-12-16.21-17-56.mp4

Architecture

This is inspired by the llm plugin system.

A plugin is just a Python package that uses the hook system from pluggy that will be installed into the same virtual env as Scope (there is a single virtual env). It can be installed via uv run daydream-scope install and uninstalled via uv run daydream-scope uninstall. These commands are thin wrappers around uv pip install and uv pip uninstall allowing us to just rely on typical Python package management and we inherit all the features of uv pip. We use uv pip instead of uv add or uv remove because the latter would modify the project deps, but we want plugins to be installed into the virtual env without changing the project deps.

A plugin adheres to ScopeHookSpec. At the moment, it has a single hookspec which is register_pipelines(). A plugin must implement this method and decorate it with @scope.core.hookimpl (example). The register_pipelines(0 method is responsible for registering one or more pipelines that should be available in Scope.

In addition to the install and uninstall commands we also introduce:

uv run daydream-scope plugins to list installed plugins
uv run daydream-scope pipelines to list available pipelines

These commands are all implemented using click.

The PipelineRegistry includes all built-in and plugin pipelines.

I've added a simple example of a plugin here that can be installed via:

uv run daydream-scope install git+https://github.com/yondonfu/scope-test-generator.git

22d559a also includes frontend changes to:

Hide Prompts box if the pipeline does not support prompts eg passthrough
Fixes an issue where you couldn't press the rewind button if the pipeline does not support prompts
Moves all the pipeline metadata that was hardcoded in the frontend into the backend as pipeline configs so that that becomes the single source of truth

Missing

There is no easy, streamlined path for tying model downloads with pipelines registered by a plugin. This is to be addressed separately.

yondonfu · 2025-12-17T02:43:34Z

pyproject.toml

 Issues = "https://github.com/daydreamlive/scope/issues"

+[tool.uv]
+preview = true


This silences a warning from uv about extra-build-dependencies usage. I added this because I wanted the CLI output when using commands to be cleaner.

yondonfu · 2025-12-17T02:44:01Z

src/scope/server/app.py

+def plugins():
+    """List all installed plugins."""
+
+    @suppress_init_output


The use of this decorator is to suppress logs when using these commands to maintain cleaner CLI output.

yondonfu · 2025-12-17T02:44:38Z

src/scope/server/app.py

-        print_version_info()
-        sys.exit(0)
+    from scope.core.pipelines.registry import (
+        PipelineRegistry,  # noqa: F401 - imported for side effects (registry initialization)


The PipelineRegistry registers pipelines at import time right now so this is just to avoid unused import errors/warnings while still ensuring that the pipeline registration happens on server startup.

yondonfu · 2025-12-17T02:45:49Z

frontend/src/pages/StreamPage.tsx

      setTransitionSteps(defaultSteps);
+
+      // Clear prompts if pipeline doesn't support them
+      if (pipeline.supportsPrompts === false) {


This is a bit of a hack. If the pipeline does not support prompts we just set the prompt to an empty string for now so that the timeline still works, but it doesn't show any text. This felt like the minimal change to get things working without requiring a larger refactor right now.

How about hiding the timeline? I think that the timeline does not bring any value if you don't have prompts.

I'm inclined to leave it for now to minimize drastic visual layout changes here and there are other things that having an indication of time progression may be useful for.

yondonfu · 2025-12-17T02:46:47Z

frontend/src/components/SettingsPanel.tsx

                        </Tooltip>
                      </TooltipProvider>
                    )}
-                    {currentPipeline.modified && (


I don't think the modified attribute was adding any value so opted to just remove to simplify since we were moving a bunch of attributes to the backend.

src/scope/core/pipelines/registry.py

leszko

Great work @yondonfu — this will significantly improve Scope’s extensibility for both us and the community 🚀

Thoughts / suggestions (ordered by priority):

Non-GPU support
This currently doesn’t work on non-GPU setups. I added an inline comment with suggested fix(es).
Separate dependencies per pipeline
We should consider running each pipeline in a separate process (similar to what I started in #173). This would allow each pipeline to have its own dependency set and completely avoid dependency conflicts.
This would be a killer feature - e.g., work like @BuffMcBigHuge’s “PersonaLive!” seems largely about resolving dependency clashes. Isolating pipelines would dramatically speed up adding new ones and aligns with where we want to go anyway.
Model downloads inside pipelines
Since you already mentioned this: it might be a good time to move model downloading into the pipeline itself. A pipeline plugin without model management feels incomplete and less useful in practice.
Existing pipelines as plugins
What do you think about migrating our existing pipelines to plugins as well? They could be preinstalled, but this would give us consistent behavior across all pipelines and allow users to uninstall default ones if desired.
Plugins vs. pipelines
Do we need both concepts? Perhaps we could simplify by having a single concept (e.g., “plugins”), with each plugin containing exactly one pipeline. No strong opinion - just something to consider.

leszko · 2025-12-17T10:00:16Z

frontend/src/pages/StreamPage.tsx

      setTransitionSteps(defaultSteps);
+
+      // Clear prompts if pipeline doesn't support them
+      if (pipeline.supportsPrompts === false) {


How about hiding the timeline? I think that the timeline does not bring any value if you don't have prompts.

leszko · 2025-12-17T10:28:45Z

src/scope/core/plugins/hookspecs.py

+    """Hook specifications for Scope plugins."""
+
+    @hookspec
+    def register_pipelines(self, register):


Do we need to force people to annotate @scope.core.hookimpl in their plugins? Maybe we should just not annotate anything, just when someone installs a plugin, then look into their code and import everything that implements our Pipeline interface.

This is required for using pluggy which relies on the decorator for detecting the hook impl. I'm inclined to just start with that unless we feel like we have a strong reason to start handrolling something ourselves. But, we can always make improvements on this if we need to.

yondonfu · 2025-12-18T00:30:17Z

A couple additional changes:

7654e86 moves heavy model related imports into init for built-in pipelines. I found this to be the simplest way to avoid triggering those imports when the pipelines are registered - those imports can be a bit slow so better to trigger them as lazily as possible.

6ab629f I noticed a frontend bug where I couldn't change the Input mode after switching to a pipeline. I think the logic flaw was that whenever the input mode changed it would check if the input mode was different than the default and then set the default. I'm not sure when this bug was introduced, but figured would push a quick fix.

yondonfu · 2025-12-18T00:40:36Z

@leszko

Non-GPU support

Addressed.

Separate dependencies per pipeline

I'm leaning towards starting off with shared venv.

A few concerns I have about separate venvs per plugin:

Will prematurely introduce complexity quickly before we've even begun working with folks to start developing plugins. It already requires some form of IPC. And when we do pipeline composition across plugins we would need something like CUDA IPC to keep data in VRAM (exiting just to move to the other pipeline would be bad for throughput/latency).
It requires more more scaffolding to provide a good UX/DX because testing the plugin with Scope is no longer just a matter of running it like a normal Python package.

I know that dep conflicts are a challenge. However, we should consider:

uv can take care of a lot dep resolution and any issues with broken venvs could be prevented with an additional dependency conflict check [1].
Given 1, if there is clear guidance for plugin devs to use version ranges when possible uv can take care of finding the appropriate versions. Many headaches here are caused if the plugin pins a specific version leaving little room for the package management to resolve against other dep declarations.
The most popular ML deps like torch generally have good policies around backwards compat so it should be relatively rare for a small difference in version to lead to major breakage. This can still happen, but the point is that it is not the default common case.

I suspect many issues can be addressed with the above. And there will remain some that cannot and the plugin devs will just need to fix their deps. My instinct is that that number is not large enough to justify immediately taking on the complexity of separate venvs per plugin. This doesn't preclude adding that as an option later.

[1] I think one of (not the only) reasons that handling deps for plugins is a pain is they are incremental installs (eg using uv pip install) into the same shared venv. As a result, you can end up in a situation where you install plugin A (eg package) and then plugin B but end up with a dep version that works with B but not with A. But, it is possible (with a reasonable # of LOC) to detect this incompatibility if you aggregate plugins as they are installed and before finalizing an install you can use uv sync or uv pip compile to make sure that all the plugins + deps are compatible with the deps in the project. This doesn't make dep conflicts go away, but it does avoid broken deps by detecting conflicts early.

Model downloads inside pipelines

I'll treat that as something to be addressed next (probably outside of this PR) and want to decide on the shared venv vs separate venv first and test on that.

Existing pipelines as plugins

I'm inclined to address this later and focus on plugin UX/DX first.

Plugins vs. pipelines

I see them as distinct concepts. A plugin can register one or many pipelines and this design leaves room for it introducing other extensions.

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

…for prompts box Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

Signed-off-by: Rafał Leszko <rafal@livepeer.org> Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

yondonfu commented Dec 17, 2025

View reviewed changes

yondonfu requested review from leszko and ryanontheinside December 17, 2025 02:48

leszko reviewed Dec 17, 2025

View reviewed changes

src/scope/core/pipelines/registry.py Show resolved Hide resolved

leszko reviewed Dec 17, 2025

View reviewed changes

yondonfu force-pushed the yf/plugins branch from b2e3446 to 6ab629f Compare December 18, 2025 00:24

yondonfu marked this pull request as draft December 21, 2025 22:26

yondonfu and others added 11 commits December 22, 2025 10:43

Add plugins module

cbf65ca

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

Support loading plugin pipelines in PipelineManager

3e50419

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

Incl plugin pipelines when listing from server

7f60d48

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

Rely on server for pipeline list in frontend

d1e914f

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

Add plugins, pipelines CLI cmd with click

5e9c447

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

Add install, uninstall (plugins) cmd in CLI

e245628

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

PipelineRegistry on server startup

955bf81

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

Rely on backend for all pipeline metadata and add conditional render …

7c8b859

…for prompts box Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

Use default pipeline only if it meets the hardware requirements

bd78ab7

Signed-off-by: Rafał Leszko <rafal@livepeer.org> Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

Lazy import heavy model imports in pipeline init

47ac7f5

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

Fix input mode bug when switching pipelines

ceba9d0

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

yondonfu force-pushed the yf/plugins branch from 6ab629f to ceba9d0 Compare December 22, 2025 15:43

leszko mentioned this pull request Dec 23, 2025

PoC: Plugins in separate environments #277

Draft

Add initial support for plugins #241

Are you sure you want to change the base?

Add initial support for plugins #241

Uh oh!

Conversation

yondonfu commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Demo

Architecture

Missing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

leszko left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yondonfu commented Dec 18, 2025

Uh oh!

yondonfu commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yondonfu commented Dec 17, 2025 •

edited

Loading

yondonfu commented Dec 18, 2025 •

edited

Loading