-
Notifications
You must be signed in to change notification settings - Fork 18
Add initial support for plugins #241
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
yondonfu
wants to merge
11
commits into
main
Choose a base branch
from
yf/plugins
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
cbf65ca
Add plugins module
yondonfu 3e50419
Support loading plugin pipelines in PipelineManager
yondonfu 7f60d48
Incl plugin pipelines when listing from server
yondonfu d1e914f
Rely on server for pipeline list in frontend
yondonfu 5e9c447
Add plugins, pipelines CLI cmd with click
yondonfu e245628
Add install, uninstall (plugins) cmd in CLI
yondonfu 955bf81
PipelineRegistry on server startup
yondonfu 7c8b859
Rely on backend for all pipeline metadata and add conditional render …
yondonfu bd78ab7
Use default pipeline only if it meets the hardware requirements
leszko 47ac7f5
Lazy import heavy model imports in pipeline init
yondonfu ceba9d0
Fix input mode bug when switching pipelines
yondonfu File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,133 +1,12 @@ | ||
| import type { InputMode } from "../types"; | ||
|
|
||
| // Unified default prompts by mode (not per-pipeline) | ||
| // These are used across all pipelines for consistency | ||
| // Default prompts by mode - used across all pipelines for consistency | ||
| export const DEFAULT_PROMPTS: Record<InputMode, string> = { | ||
| text: "A 3D animated scene. A **panda** walks along a path towards the camera in a park on a spring day.", | ||
| video: | ||
| "A 3D animated scene. A **panda** sitting in the grass, looking around.", | ||
| }; | ||
|
|
||
| export interface PipelineInfo { | ||
| name: string; | ||
| about: string; | ||
| docsUrl?: string; | ||
| modified?: boolean; | ||
| estimatedVram?: number; // GB | ||
| requiresModels?: boolean; // Whether this pipeline requires models to be downloaded | ||
| defaultTemporalInterpolationMethod?: "linear" | "slerp"; // Default method for temporal interpolation | ||
| defaultTemporalInterpolationSteps?: number; // Default number of steps for temporal interpolation | ||
| supportsLoRA?: boolean; // Whether this pipeline supports LoRA adapters | ||
| supportsVACE?: boolean; // Whether this pipeline supports VACE (Video All-In-One Creation and Editing) | ||
|
|
||
| // Multi-mode support | ||
| supportedModes: InputMode[]; | ||
| defaultMode: InputMode; | ||
| } | ||
|
|
||
| export const PIPELINES: Record<string, PipelineInfo> = { | ||
| streamdiffusionv2: { | ||
| name: "StreamDiffusionV2", | ||
| docsUrl: | ||
| "https://github.com/daydreamlive/scope/blob/main/src/scope/core/pipelines/streamdiffusionv2/docs/usage.md", | ||
| about: | ||
| "A streaming pipeline and autoregressive video diffusion model from the creators of the original StreamDiffusion project. The model is trained using Self-Forcing on Wan2.1 1.3b with modifications to support streaming. Includes VACE (All-In-One Video Creation and Editing) for reference image conditioning and structural guidance (depth, flow, pose).", | ||
| modified: true, | ||
| estimatedVram: 20, | ||
| requiresModels: true, | ||
| defaultTemporalInterpolationMethod: "slerp", | ||
| defaultTemporalInterpolationSteps: 0, | ||
| supportsLoRA: true, | ||
| supportsVACE: true, | ||
| // Multi-mode support | ||
| supportedModes: ["text", "video"], | ||
| defaultMode: "video", | ||
| }, | ||
| longlive: { | ||
| name: "LongLive", | ||
| docsUrl: | ||
| "https://github.com/daydreamlive/scope/blob/main/src/scope/core/pipelines/longlive/docs/usage.md", | ||
| about: | ||
| "A streaming pipeline and autoregressive video diffusion model from Nvidia, MIT, HKUST, HKU and THU. The model is trained using Self-Forcing on Wan2.1 1.3b with modifications to support smoother prompt switching and improved quality over longer time periods while maintaining fast generation. Includes VACE (All-In-One Video Creation and Editing) for reference image conditioning and structural guidance (depth, flow, pose).", | ||
| modified: true, | ||
| estimatedVram: 20, | ||
| requiresModels: true, | ||
| defaultTemporalInterpolationMethod: "slerp", | ||
| defaultTemporalInterpolationSteps: 0, | ||
| supportsLoRA: true, | ||
| supportsVACE: true, | ||
| // Multi-mode support | ||
| supportedModes: ["text", "video"], | ||
| defaultMode: "text", | ||
| }, | ||
| "krea-realtime-video": { | ||
| name: "Krea Realtime Video", | ||
| docsUrl: | ||
| "https://github.com/daydreamlive/scope/blob/main/src/scope/core/pipelines/krea_realtime_video/docs/usage.md", | ||
| about: | ||
| "A streaming pipeline and autoregressive video diffusion model from Krea. The model is trained using Self-Forcing on Wan2.1 14b.", | ||
| modified: true, | ||
| estimatedVram: 32, | ||
| requiresModels: true, | ||
| defaultTemporalInterpolationMethod: "linear", | ||
| defaultTemporalInterpolationSteps: 4, | ||
| supportsLoRA: true, | ||
| // Multi-mode support | ||
| supportedModes: ["text", "video"], | ||
| defaultMode: "text", | ||
| }, | ||
| "reward-forcing": { | ||
| name: "RewardForcing", | ||
| docsUrl: | ||
| "https://github.com/daydreamlive/scope/blob/main/src/scope/core/pipelines/reward_forcing/docs/usage.md", | ||
| about: | ||
| "A streaming pipeline and autoregressive video diffusion model from ZJU, Ant Group, SIAS-ZJU, HUST and SJTU. The model is trained with Rewarded Distribution Matching Distillation using Wan2.1 1.3b as the base model. Includes VACE (All-In-One Video Creation and Editing) for reference image conditioning and structural guidance (depth, flow, pose).", | ||
| modified: true, | ||
| estimatedVram: 20, | ||
| requiresModels: true, | ||
| defaultTemporalInterpolationMethod: "slerp", | ||
| defaultTemporalInterpolationSteps: 0, | ||
| supportsLoRA: true, | ||
| supportsVACE: true, | ||
| // Multi-mode support | ||
| supportedModes: ["text", "video"], | ||
| defaultMode: "text", | ||
| }, | ||
| passthrough: { | ||
| name: "Passthrough", | ||
| about: | ||
| "A pipeline that returns the input video without any processing that is useful for testing and debugging.", | ||
| requiresModels: false, | ||
| // Video-only pipeline | ||
| supportedModes: ["video"], | ||
| defaultMode: "video", | ||
| }, | ||
| }; | ||
|
|
||
| export function pipelineSupportsLoRA(pipelineId: string): boolean { | ||
| return PIPELINES[pipelineId]?.supportsLoRA === true; | ||
| } | ||
|
|
||
| export function pipelineSupportsVACE(pipelineId: string): boolean { | ||
| return PIPELINES[pipelineId]?.supportsVACE === true; | ||
| } | ||
|
|
||
| export function pipelineSupportsMode( | ||
| pipelineId: string, | ||
| mode: InputMode | ||
| ): boolean { | ||
| return PIPELINES[pipelineId]?.supportedModes?.includes(mode) ?? false; | ||
| } | ||
|
|
||
| export function pipelineIsMultiMode(pipelineId: string): boolean { | ||
| const modes = PIPELINES[pipelineId]?.supportedModes ?? []; | ||
| return modes.length > 1; | ||
| } | ||
|
|
||
| export function getPipelineDefaultMode(pipelineId: string): InputMode { | ||
| return PIPELINES[pipelineId]?.defaultMode ?? "text"; | ||
| } | ||
|
|
||
| export function getDefaultPromptForMode(mode: InputMode): string { | ||
| return DEFAULT_PROMPTS[mode]; | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think the
modifiedattribute was adding any value so opted to just remove to simplify since we were moving a bunch of attributes to the backend.