diff --git a/.gitignore b/.gitignore index 3a7b54da..4ef710dd 100644 --- a/.gitignore +++ b/.gitignore @@ -76,3 +76,5 @@ python-embed/ # Downloaded during CI for Windows NSIS installer resources/vc_redist.x64.exe + +reference_projects/ \ No newline at end of file diff --git a/.vscode/launch.json b/.vscode/launch.json index 84ae6ae4..8e55d5d8 100644 --- a/.vscode/launch.json +++ b/.vscode/launch.json @@ -1,6 +1,17 @@ { "version": "0.2.0", "configurations": [ + { + "name": "Run: pnpm dev", + "type": "node", + "request": "launch", + "runtimeExecutable": "pnpm", + "runtimeArgs": [ + "dev" + ], + "cwd": "${workspaceFolder}", + "console": "integratedTerminal" + }, { "name": "Attach: Electron Main", "type": "node", diff --git a/backend/ltx2_server.py b/backend/ltx2_server.py index c7898f5f..78d1b9e8 100644 --- a/backend/ltx2_server.py +++ b/backend/ltx2_server.py @@ -181,9 +181,13 @@ def _resolve_force_api_generations() -> bool: return force_api_generations +import os + FORCE_API_GENERATIONS = _resolve_force_api_generations() +_BYPASS_MODEL_CHECK = os.environ.get("LTX_BYPASS_MODEL_CHECK") == "1" + REQUIRED_MODEL_TYPES: frozenset[ModelFileType] = ( - frozenset() if FORCE_API_GENERATIONS else DEFAULT_REQUIRED_MODEL_TYPES + frozenset() if (FORCE_API_GENERATIONS or _BYPASS_MODEL_CHECK) else DEFAULT_REQUIRED_MODEL_TYPES ) CAMERA_MOTION_PROMPTS = { diff --git a/backend/runtime_config/runtime_policy.py b/backend/runtime_config/runtime_policy.py index 7b84dc89..40f9312f 100644 --- a/backend/runtime_config/runtime_policy.py +++ b/backend/runtime_config/runtime_policy.py @@ -1,10 +1,14 @@ """Runtime policy decisions for forced API mode.""" from __future__ import annotations +import os def decide_force_api_generations(system: str, cuda_available: bool, vram_gb: int | None) -> bool: """Return whether API-only generation must be forced for this runtime.""" + if os.environ.get("LTX_BYPASS_API_CHECK") == "1": + return False + if system == "Darwin": return True diff --git a/change-requests/CR001-comfyui-integration/HLD.md b/change-requests/CR001-comfyui-integration/HLD.md new file mode 100644 index 00000000..f77dabe9 --- /dev/null +++ b/change-requests/CR001-comfyui-integration/HLD.md @@ -0,0 +1,57 @@ +# High-Level Design (HLD): ComfyUI Integration + +## 1. Overview + +This High-Level Design details the integration of a ComfyUI engine into the LTX-Desktop application. As determined in the Impact Assessment, this integration follows **Option B: Proxy-Based Metadata Mapping**. + +The core architectural philosophy is **Additive Isolation**: the ComfyUI integration will be built *on top* of the existing local generation capabilities without modifying their underlying logic. This minimizes merge conflicts with the upstream fork and ensures the default native GPU experience is preserved. + +## 2. Core Components + +### 2.1. App Settings and API Types (Additive) + +* **`AppSettings`**: A new setting, `generation_backend` (Literal: `"local" | "comfyui"`), will be added to dictate the routing logic. +* **`api_types.py`**: Generation request payloads (e.g., `VideoGenerationRequest`) will be extended with an optional `workflow_params: dict[str, Any] | None` to pass dynamic proxy widget values from the UI to the backend. + +### 2.2. State Management (`AppState`) + +To avoid disrupting the highly tuned local `GpuSlot` management: +* A new state slot, **`ComfyUIJobSlot`**, will be introduced in `AppState`. +* The centralized `AppHandler` lock will protect this new slot exactly as it protects the `GpuSlot`. + +### 2.3. ComfyUI Service Module + +A new, isolated module (`backend/services/comfyui/`) will encapsulate all ComfyUI-specific logic: + +1. **`WorkflowParser`**: + * Reads predefined ComfyUI JSON workflows. + * Extracts the `proxyWidgets` metadata to identify which internal node parameters are exposed to the UI. +2. **`ComfyUIClient`**: + * Handles HTTP communication with the ComfyUI server (e.g., `/prompt`, `/upload/image`, `/history`). + * Manages WebSocket connections (if required) for real-time progress updates. +3. **`ComfyUIPipelineAdapters`**: + * Implements the existing strictly-typed protocols (e.g., `FastVideoPipeline`). + * Translates the incoming `VideoGenerationRequest` (including `workflow_params`) into the final execution graph JSON. + +### 2.4. Generation Handler Routing + +The `GenerationHandler` will act as a router based on the `generation_backend` setting: + +* **If `"local"`**: The handler proceeds normally, acquiring the `GpuSlot` and delegating to the native `services.video_processor`. +* **If `"comfyui"`**: The handler bypasses the `GpuSlot`, acquires the `ComfyUIJobSlot`, and delegates to the `ComfyUIPipelineAdapter`. + +### 2.5. Progress Translation + +To ensure the frontend requires zero changes to its progress tracking logic: +* The `ComfyUIPipelineAdapter` will spawn a background polling task (using the existing `TaskRunner`). +* This task will translate ComfyUI's native execution progress into the exact `GenerationProgress` (e.g., `GenerationRunning`, `GenerationComplete`) state objects expected by `AppState`. + +## 3. Architectural Flow (ComfyUI Active) + +1. **UI Configuration**: Frontend fetches available workflows via a new endpoint (parsed by `WorkflowParser`) and dynamically renders controls for the exposed `proxyWidgets`. +2. **Submission**: User clicks generate. Frontend sends `VideoGenerationRequest` including `workflow_params`. +3. **Routing**: `GenerationHandler` sees `generation_backend == "comfyui"`. +4. **Locking**: Handler acquires lock -> sets `ComfyUIJobSlot` to running -> unlocks. +5. **Execution**: `ComfyUIPipelineAdapter` constructs the final JSON graph and sends it to the `ComfyUIClient`. +6. **Progress**: Background task polls ComfyUI, locking briefly to update `ComfyUIJobSlot` progress. +7. **Completion**: Adapter retrieves the final media from ComfyUI, saves it locally, and updates state to `GenerationComplete`. diff --git a/change-requests/CR001-comfyui-integration/IMPLEMENTATION_PLAN.md b/change-requests/CR001-comfyui-integration/IMPLEMENTATION_PLAN.md new file mode 100644 index 00000000..11e0be68 --- /dev/null +++ b/change-requests/CR001-comfyui-integration/IMPLEMENTATION_PLAN.md @@ -0,0 +1,76 @@ +# Implementation Plan: ComfyUI Integration + +This plan outlines the steps required to implement the ComfyUI integration described in the High-Level Design (HLD), adhering to the principles of "Additive Isolation" to minimize fork impact. + +## Phase 1: Foundation (Settings & API Types) + +**Goal:** Extend the core data structures to support routing and dynamic payload parameters without breaking existing schemas. + +1. **Update Settings:** + * Modify `settings.json` and `backend/state/app_settings.py` to add `generation_backend` (defaulting to `"local"`). +2. **Update API Types:** + * In `backend/api_types.py`, add `workflow_params: dict[str, Any] | None = None` to generation requests (e.g., `VideoGenerationRequest`). +3. **Update State Types:** + * In `backend/state/app_state_types.py`, define a new `ComfyUIJobSlot` (tracking status, progress, current job ID). + * Add `comfyui_job: ComfyUIJobSlot | None` to the root `AppState` definition. + +## Phase 2: Core ComfyUI Services + +**Goal:** Create the isolated module (`backend/services/comfyui/`) for parsing workflows and communicating with the ComfyUI server. + +1. **Create Service Directory:** Initialize `backend/services/comfyui/`. +2. **Implement `WorkflowParser`:** + * Create logic to read JSON workflows from a designated directory. + * Extract `proxyWidgets` metadata from node properties. +3. **Implement `ComfyUIClient`:** + * Create an asynchronous HTTP client to communicate with the ComfyUI API (`/prompt`, `/upload/image`, `/history`, `/view`). +4. **Expose Workflows Endpoint:** + * Create a new route in `backend/_routes/` (e.g., `workflows.py`) to expose the parsed workflows and their configurable parameters to the frontend. + * Wire the route into `app_factory.py`. + +## Phase 3: Pipeline Adapters and Progress Tracking + +**Goal:** Implement the adapter that bridges the strictly typed LTX-Desktop protocols with the dynamic ComfyUI engine. + +1. **Implement `ComfyUIVideoPipeline`:** + * Create a class implementing the `FastVideoPipeline` (or relevant) protocol. + * Implement graph construction: merge the base JSON workflow with the incoming `workflow_params`. +2. **Implement Background Polling:** + * Use the existing `TaskRunner` to spawn a background task upon job submission. + * Poll the ComfyUI server for progress on the submitted `prompt_id`. + * Translate the progress into standard `GenerationProgress` state objects. + +## Phase 4: Handler Routing & Locking Integration + +**Goal:** Update the centralized handler to securely route tasks based on the active backend. + +1. **Update `GenerationHandler`:** + * In `backend/handlers/generation_handler.py`, read `generation_backend` from settings. + * Implement branching logic: + * If `"local"`: Use `GpuSlot` and standard pipeline logic (existing code). + * If `"comfyui"`: Acquire lock, validate `ComfyUIJobSlot` is idle, set to running, release lock, and dispatch to `ComfyUIVideoPipeline`. +2. **Ensure Lock Safety:** + * Verify the "lock -> check -> unlock -> heavy work -> lock -> update" pattern is strictly followed for the new `ComfyUIJobSlot`. + +## Phase 5: Frontend Integration + +**Goal:** Update the React frontend to dynamically render UI elements based on the parsed ComfyUI workflows. + +1. **Backend Toggle:** Add a UI toggle in the settings to switch between Local and ComfyUI backends. +2. **Fetch Workflows:** On mount (if ComfyUI is active), fetch the available workflows from the new backend endpoint. +3. **Dynamic Rendering:** + * Parse the returned proxy widget schemas. + * Dynamically render sliders, dropdowns, and text inputs based on the expected types of the proxy widgets. +4. **Submission Logic:** Update the `backendFetch` calls for generation to include the user-configured `workflow_params` dictionary. + +## Phase 6: Testing and Validation + +**Goal:** Ensure the integration is robust and the local pipeline remains unaffected. + +1. **Backend Integration Tests:** + * Create new tests in `backend/tests/` using fakes for the `ComfyUIClient`. + * Verify routing logic works correctly based on settings. +2. **Type Checking:** + * Run `pnpm typecheck` to ensure the new dynamic dictionaries haven't violated strict mode rules elsewhere. +3. **Local Regression:** + * Run existing `backend:test` suite to guarantee standard local generation is completely isolated and functional. \ No newline at end of file diff --git a/change-requests/CR001-comfyui-integration/docs/comfyui-integration-impact.md b/change-requests/CR001-comfyui-integration/docs/comfyui-integration-impact.md new file mode 100644 index 00000000..3221d984 --- /dev/null +++ b/change-requests/CR001-comfyui-integration/docs/comfyui-integration-impact.md @@ -0,0 +1,94 @@ +# Impact Assessment: Replacing Backend Generation with Configurable ComfyUI Workflows + +## 1. Executive Summary + +This document assesses the architectural impact of integrating a ComfyUI backend alongside the current local generative pipelines. The goal is to allow dynamic, runtime switching between the existing local GPU implementations and a configurable ComfyUI workflow engine. + +Based on recent analysis of reference projects (Krita AI Diffusion and Vlo), this document focuses on clarifying the architectural options for procedural node discovery and UI mapping. The objective of this phase is to review these options prior to committing to a final architectural decision. + +--- + +## 2. Current Architecture Overview + +The backend uses a local FastAPI server where endpoints delegate business logic to a centralized `AppHandler`. + +- **State Management:** A highly normalized, typed `AppState` manages limited resources (e.g., `GpuSlot`, `CpuSlot`, `DownloadingSession`). +- **Concurrency & Locking:** A single shared `RLock` protects `AppState`. Handlers follow a strict "lock -> check -> unlock -> heavy work -> lock -> update" pattern to prevent blocking the server during generation. +- **Service Boundaries:** Heavy generative tasks are isolated behind strictly typed Python Protocols in `backend/services/` (e.g., `FastVideoPipeline`). +- **Generation Lifecycle:** `GenerationHandler` tracks progress using normalized state machines (`GenerationRunning`, `GenerationComplete`, etc.). + +--- + +## 3. Proposed Architecture Options for UI-to-Node Mapping + +A core challenge is how the frontend UI (sliders, dropdowns for models/LoRAs) dynamically maps to and controls the underlying ComfyUI node graph. Two distinct architectural options have been identified based on industry reference projects. + +### Option A: Functional Mapping via Dedicated Custom Nodes (The Krita Approach) + +In this approach, the integration relies on abstracting the low-level node graph into a high-level Python API, tightly coupled with a dedicated suite of custom ComfyUI nodes. + +* **Mechanism:** + * Relies on the ComfyUI `/object_info` API to discover available nodes and their schemas. + * The backend implements a "Builder Pattern" (`ComfyWorkflow`) that translates high-level UI requests (e.g., `generate_video(prompt, seed)`) into specific node instantiations. + * **Crucial Prerequisite:** Requires installing a dedicated ComfyUI extension with custom nodes (e.g., `LTX_LoadImage`, `LTX_InjectVideo`) designed specifically to bridge the communication gap (e.g., handling in-memory transfers or specific app logic). +* **Pros:** + * **Strong Type Safety:** The builder validates inputs/outputs against the `object_info` schema before execution. + * **Simpler UI Logic:** The UI only interacts with high-level parameters; the backend handles the complex graph construction. +* **Cons:** + * **High Maintenance:** Tightly coupled to specific custom nodes. Any changes to the node logic require updating the backend builder. + * **Less Flexible:** Harder for users to drop in arbitrary, wildly different ComfyUI workflows without backend updates. + +### Option B: Proxy-Based Metadata Mapping (The Vlo Approach) + +This approach is workflow-agnostic and relies on embedding UI mapping rules directly within the ComfyUI workflow JSON metadata. + +* **Mechanism:** + * Relies on a custom metadata field, specifically the established ComfyUI convention `proxyWidgets`, located within the `properties` dictionary of Subgraphs (Group Nodes) or individual nodes. + * The workflow JSON explicitly defines mapping tuples (e.g., `["node_id_31", "seed"]`). + * The backend parses these JSON files at startup, discovers the exposed parameters, and serves this dynamic schema to the frontend. + * **Crucial Prerequisite:** Requires zero dedicated custom nodes. It interfaces with standard ComfyUI nodes and relies entirely on the JSON metadata structure. +* **Pros:** + * **Highly Decoupled & Workflow Agnostic:** The UI structure is defined *by* the graph. Users can drop in entirely new workflows (using standard nodes) as long as they tag the inputs with `proxyWidgets`. + * **Minimal Backend Logic:** The backend acts primarily as a pass-through and normalizer, rather than a complex graph builder. +* **Cons:** + * **Weaker Typing:** Relies on dynamic dictionaries passing through the backend, requiring careful validation logic. + * **Complex JSON Maintenance:** The burden of defining the UI shifts to whoever authors the ComfyUI JSON workflows; they must correctly set up the `proxyWidgets` arrays. + +--- + +## 4. Architectural Impact on LTX-Desktop Backend (Option B Selected) + +Following review, **Option B (Proxy-Based Metadata Mapping)** has been selected. The primary directive for this integration is to **maximize out-of-the-box compatibility** while ensuring a **minimal fork update impact**. + +Crucially, the ComfyUI integration must be added *on top* of existing services. Current LTX Desktop functionality (local, native GPU generation) must be fully retained and operate exactly as before when ComfyUI is not active. By isolating the new ComfyUI logic, we ensure that upstream merges from the original `LTX-Desktop` repository remain trivial. + +### 4.1. Core Principle: Isolation for Minimal Merge Conflicts +To minimize merge conflicts when pulling from the upstream fork, the ComfyUI integration will avoid heavily modifying existing core files (like `app_handler.py` or complex state machines) wherever possible. Instead, it will rely on new interface implementations and isolated modules. + +* **New Modules:** All ComfyUI-specific parsing, proxy mapping, and network communication will live in a strictly separated directory (e.g., `backend/services/comfyui/`). +* **Interface Implementation:** The ComfyUI engine will implement the existing pipeline interfaces (e.g., creating a `ComfyUIVideoPipeline` that adheres to the same Protocol as `FastVideoPipeline`). This allows the core `GenerationHandler` to treat it as just another backend without knowing its internal complexities. + +### 4.2. AppSettings and API Types (Additive Changes) +Changes to core API files will be strictly additive, preserving all existing schemas. +* **`app_settings.py`:** Add a new, optional `generation_backend` flag (defaulting to `local`). +* **`api_types.py`:** Add a new, optional `workflow_params: dict[str, Any] | None = None` to the generation request models to handle the dynamic `proxyWidgets` inputs. Existing strictly-typed fields (prompt, seed, etc.) remain untouched and can be mapped internally if ComfyUI is the active backend. + +### 4.3. State Management (`AppState` & `GpuSlot`) +The existing `GpuSlot` logic, which carefully manages local VRAM, must remain untouched to ensure the default local generation experience is not compromised. +* **Additive State:** A new, distinct slot (e.g., `ExternalSlot` or `ComfyUIJobSlot`) will be introduced to `AppState`. +* **Locking:** The `AppHandler` will use the same locking mechanism to protect this new slot, ensuring thread safety without needing to rewrite the existing local GPU locking logic. If the active backend is ComfyUI, the handler checks the `ExternalSlot` instead of the `GpuSlot`. + +### 4.4. Generation Handler and Progress Tracking +The existing `GenerationHandler` relies heavily on the local process actively reporting progress. +* **Adapter Pattern:** A `ComfyUIAdapter` service will act as a bridge. When a generation task is dispatched to ComfyUI, the adapter will use the `TaskRunner` to spawn a background polling task. +* **State Translation:** This polling task will fetch ComfyUI's native progress (using its API/Websocket) and translate it into the *exact same* `GenerationProgress` state objects currently expected by the frontend. This ensures the frontend UI requires minimal, if any, modifications to display progress bars. + +--- + +## 5. Conclusion + +By selecting **Option B (Proxy-Based Metadata Mapping)**, we achieve a highly flexible, workflow-agnostic system. + +By applying a strict **"Additive Isolation"** architectural lens, we ensure that: +1. **Original functionality is preserved:** Local generation remains completely unaffected and acts as the default. +2. **Upstream merges are trivial:** By avoiding modifications to core logic loops and instead adding new interface implementations and isolated service folders, we minimize the "fork divergence," allowing the project to easily consume future updates from the original Lightricks repository. \ No newline at end of file diff --git a/change-requests/CR001-comfyui-integration/docs/krita-ai-diffusion-technique.md b/change-requests/CR001-comfyui-integration/docs/krita-ai-diffusion-technique.md new file mode 100644 index 00000000..32c064b6 --- /dev/null +++ b/change-requests/CR001-comfyui-integration/docs/krita-ai-diffusion-technique.md @@ -0,0 +1,35 @@ +# Procedural ComfyUI Node Discovery and UI Mapping in Krita AI Diffusion + +## Overview + +Krita AI Diffusion manages ComfyUI integration by abstracting the low-level node graph into a high-level Python API. This approach ensures type safety and simplifies the construction of complex workflows while remaining flexible enough to support custom nodes. + +## Prerequisites + +This technique requires the installation of a **dedicated ComfyUI extension** (`ComfyUI-Krita-AI-Diffusion`). + +- **Custom Nodes**: The system relies on specific nodes (prefixed with `ETN_`, such as `ETN_LoadImageCache`, `ETN_InjectImage`, and `ETN_LoadMaskBase64`) to handle specialized tasks like in-memory image transfer and precise canvas synchronization. +- **Fixed API Contract**: The Python backend is tightly coupled to these custom nodes. Without them, the high-level builder cannot generate the necessary graph structure to communicate with Krita. + +## Node Discovery Technique + +The system relies on the `/object_info` endpoint of the ComfyUI API to procedurally discover available nodes and their respective input/output schemas. + +1. **Schema Retrieval**: Upon connection, the client fetches the full `object_info` dictionary. +2. **Type Mapping**: It maps ComfyUI types (e.g., `IMAGE`, `LATENT`, `MODEL`) to internal Python classes. +3. **Validation**: The `ComfyWorkflow` builder uses this schema to validate inputs and outputs at construction time, preventing the creation of invalid graphs before they are even sent to the server. + +## UI-to-Node Mapping + +Unlike systems that use a 1:1 visual mapping, Krita AI Diffusion uses a **Functional Mapping** approach: + +1. **Abstract Workflows**: The UI interacts with a `Workflow` class that defines high-level operations (e.g., `sampling`, `upscaling`). +2. **Builder Pattern**: These high-level operations are translated into a sequence of `ComfyWorkflow` method calls (e.g., `w.load_checkpoint()`, `w.ksampler()`). +3. **Dynamic Node IDs**: Node IDs are generated procedurally during the building process. The UI does not need to know specific node IDs; it only cares about the parameters (e.g., `seed`, `steps`, `denoise`) passed to the high-level API. +4. **Custom Workflows**: For user-provided workflows, the system imports the JSON graph and uses the `object_info` schema to identify "input" nodes (e.g., `PrimitiveNode`, `CheckpointLoaderSimple`) that should be exposed as UI widgets based on their titles or specific metadata. + +## Key Advantages + +- **Type Safety**: Prevents common "wrong input type" errors. +- **Maintainability**: Changes in the underlying ComfyUI node names can be handled by updating the mapping logic in one place. +- **User Experience**: Simplifies complex graphs into a familiar, tool-like interface for artists. diff --git a/change-requests/CR001-comfyui-integration/docs/vlo-technique.md b/change-requests/CR001-comfyui-integration/docs/vlo-technique.md new file mode 100644 index 00000000..7f88b8bd --- /dev/null +++ b/change-requests/CR001-comfyui-integration/docs/vlo-technique.md @@ -0,0 +1,43 @@ +# Procedural ComfyUI Node Discovery and UI Mapping in Vlo + +## Overview + +Vlo implements a "Proxy-Based" approach to ComfyUI node management. It allows for a highly flexible UI that can dynamically adapt to various underlying ComfyUI workflows by explicitly defining links between UI elements and backend nodes within the workflow metadata itself. + +## Prerequisites + +Vlo is **workflow-agnostic** and does not require dedicated custom nodes for its core mapping functionality. + +- **Standard Nodes**: It interfaces with standard ComfyUI nodes (e.g., `LoadImage`, `CLIPTextEncode`, `KSampler`) and popular community extensions (e.g., `VideoHelperSuite`) using their default schemas. +- **Metadata-Driven**: The only requirement is the inclusion of specific metadata (`proxyWidgets`) within the JSON workflow file's `properties` field. This allows the system to remain compatible with any node as long as the mapping is explicitly defined in the graph. + +## Node Discovery Technique + +Vlo's discovery is centered around the **Workflow Metadata** and the `properties` field of ComfyUI nodes. + +1. **Graph Introspection**: The system parses the ComfyUI JSON workflow (both API and Graph formats). +2. **Property Extraction**: It specifically looks for a custom `proxyWidgets` field within the `properties` dictionary of a node. +3. **State Synchronization**: The backend maintains a cache of the current graph state and uses it to resolve which UI-exposed parameters correspond to which internal node inputs. + +## UI-to-Node Mapping + +Vlo uses a **Proxy Widget Mapping** technique to bridge the UI and the backend: + +1. **Explicit Mapping**: The `proxyWidgets` property contains a list of mappings. Each entry defines a `target_node_id` and a `target_param_name`. +2. **Virtual Groups**: UI elements are organized into "Groups" (e.g., "Sampling Settings", "Model Selection") based on the `group_id` and `group_title` associated with the proxy mapping. +3. **Procedural UI Generation**: The frontend receives a list of these groups and their associated controls. When a user modifies a value in the UI, the backend uses the mapping to update the specific node and parameter in the graph before submission. +4. **Node Normalization**: Before sending the final graph to ComfyUI, the system "normalizes" it, ensuring that all proxy-driven changes are correctly applied to the final execution graph. + +## Key Advantages + +- **Decoupling**: The UI structure is not hardcoded to a specific node graph; it is defined *by* the graph. +- **Granular Control**: Specific parameters from different nodes can be grouped together in the UI, even if they are far apart in the actual graph. +- **Workflow Agnostic**: Any ComfyUI workflow can be "UI-enabled" simply by adding the appropriate `proxyWidgets` metadata to the nodes. + +## Findings on `proxyWidgets` Convention + +Research indicates that `proxyWidgets` is an established **ComfyUI metadata convention** used specifically for **Subgraphs (Group Nodes)** and **Blueprints**. + +1. **Standard Purpose**: In the broader ComfyUI ecosystem, `proxyWidgets` allows a "Group Node" or "Subgraph" to expose internal parameters (like `seed`, `steps`, or `prompt`) to the top-level UI of the collapsed group. +2. **Implementation**: It is stored in the `properties` field of a node in the workflow JSON as an array of tuples: `["internal_node_id", "widget_name"]`. +3. **Vlo's Leverage**: Vlo utilizes this standard mapping logic to drive its entire dynamic UI generation. By following this convention, Vlo can interface with any complex workflow that has been organized into logical groups/subgraphs without needing custom backend code for every new node type. diff --git a/change-requests/CR002-developer-configuration/HLD.md b/change-requests/CR002-developer-configuration/HLD.md new file mode 100644 index 00000000..385cac31 --- /dev/null +++ b/change-requests/CR002-developer-configuration/HLD.md @@ -0,0 +1,54 @@ +# High-Level Design (HLD): Developer Configuration - API Check Bypass + +## 1. Overview + +This document outlines the changes made for **CR002: Developer Configuration** to streamline the local development experience. The primary goal of this iteration is to allow developers to easily bypass the mandatory "Connect API Keys" gate that blocks access to the application UI on machines lacking sufficient hardware or proper setup. + +The solution ensures that developers can access and work on the application UI (e.g., for integrating ComfyUI workflows) without needing actual API keys or a production-ready local GPU environment, while strictly maintaining the integrity of the production codebase. + +## 2. Core Components + +### 2.1. Backend Runtime Policy & Model Checking + +The application normally enforces two major gates before allowing access to the main UI: +1. **API Key Gate:** Triggered if the system lacks the necessary hardware to run local models (managed by `decide_force_api_generations`). +2. **Model Download Gate:** Triggered if local generation is allowed but the required model files (e.g., checkpoints) are missing on disk. + +To bypass these gates, two environment variables were introduced: + +* **`LTX_BYPASS_API_CHECK`:** + * **Location:** `backend/runtime_config/runtime_policy.py` (`decide_force_api_generations`) + * **Logic:** If set to `"1"`, the function returns `False`, tricking the app into believing local generation is fully supported, thereby dismissing the mandatory API key prompt. +* **`LTX_BYPASS_MODEL_CHECK`:** + * **Locations:** + * Backend: `backend/ltx2_server.py` + * Electron Main Process: `electron/ipc/app-handlers.ts` (`getSetupStatus`) + * **Logic:** + * In the backend, if set to `"1"`, the `REQUIRED_MODEL_TYPES` frozen set is overridden to be empty (`frozenset()`). This tricks the frontend's model status check (`/api/models/status`) into believing all necessary models are already downloaded. + * In the Electron main process, if set to `"1"`, the IPC handler immediately returns `{ needsSetup: false, needsLicense: false }`, overriding the local `app_state.json` file. + * **Impact:** Together, these bypass both the local app state checks and the backend filesystem checks, completely skipping the "Choose Location" installation screen. + +### 2.2. Development Scripts configuration + +To ensure these bypasses are seamless and strictly limited to local development, the environment variables are injected via `package.json` scripts rather than relying on a global `.env` file. + +* **Change:** The `dev` and `dev:debug` scripts in `package.json` were updated using `cross-env`. +* **Old Scripts:** + ```json + "dev": "vite", + "dev:debug": "cross-env BACKEND_DEBUG=1 ELECTRON_DEBUG=1 vite" + ``` +* **New Scripts:** + ```json + "dev": "cross-env LTX_BYPASS_API_CHECK=1 LTX_BYPASS_MODEL_CHECK=1 vite", + "dev:debug": "cross-env BACKEND_DEBUG=1 ELECTRON_DEBUG=1 LTX_BYPASS_API_CHECK=1 LTX_BYPASS_MODEL_CHECK=1 vite" + ``` + +## 3. Security and Production Safety + +* **No Global `.env`:** By avoiding a `.env` file, we prevent accidental commits or persistent global state that could inadvertently disable the API or model checks outside of an active `pnpm dev` session. +* **Production Build Isolation:** The `build`, `build:skip-python`, and `build:fast` scripts in `package.json` do **not** include these bypass flags. Consequently, packaged release binaries remain secure and will correctly enforce hardware checks, API key requirements, and model downloads for end-users. + +## 4. Conclusion + +This non-invasive approach provides a frictionless path for frontend and integration development without compromising the strict hardware and licensing gates required for the application's production release. \ No newline at end of file diff --git a/electron/ipc/app-handlers.ts b/electron/ipc/app-handlers.ts index f7eb3d27..e56ffe57 100644 --- a/electron/ipc/app-handlers.ts +++ b/electron/ipc/app-handlers.ts @@ -16,6 +16,10 @@ function getModelsPath(): string { } function getSetupStatus(settingsPath: string): { needsSetup: boolean; needsLicense: boolean } { + if (process.env.LTX_BYPASS_MODEL_CHECK === '1') { + return { needsSetup: false, needsLicense: false } + } + if (!fs.existsSync(settingsPath)) { return { needsSetup: true, needsLicense: true } } diff --git a/package.json b/package.json index 2204df1a..be06b5a0 100644 --- a/package.json +++ b/package.json @@ -14,8 +14,8 @@ "packageManager": "pnpm@10.30.3", "scripts": { "setup:dev": "node scripts/run-script.js scripts/setup-dev", - "dev": "vite", - "dev:debug": "cross-env BACKEND_DEBUG=1 ELECTRON_DEBUG=1 vite", + "dev": "cross-env LTX_BYPASS_API_CHECK=1 LTX_BYPASS_MODEL_CHECK=1 vite", + "dev:debug": "cross-env BACKEND_DEBUG=1 ELECTRON_DEBUG=1 LTX_BYPASS_API_CHECK=1 LTX_BYPASS_MODEL_CHECK=1 vite", "typecheck": "concurrently \"pnpm run typecheck:ts\" \"pnpm run typecheck:py\"", "typecheck:ts": "tsc --noEmit", "typecheck:py": "cd backend && uv run pyright",