From db297b22a9092ab835ae0bcaf9986ee59f1f6aa0 Mon Sep 17 00:00:00 2001 From: "google-labs-jules[bot]" <161369871+google-labs-jules[bot]@users.noreply.github.com> Date: Fri, 13 Mar 2026 06:34:32 +0000 Subject: [PATCH 01/12] docs: add ComfyUI integration impact assessment Creates a new document detailing the architectural changes required to replace existing local generation pipelines with a configurable ComfyUI workflow engine, evaluating both the Facade and Generic Workflow Engine approaches. This document addresses dynamic state management, workflow parsing, and locking implications without modifying existing code. Co-authored-by: adunato <7783232+adunato@users.noreply.github.com> --- docs/comfyui-integration-impact.md | 90 ++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 docs/comfyui-integration-impact.md diff --git a/docs/comfyui-integration-impact.md b/docs/comfyui-integration-impact.md new file mode 100644 index 00000000..71555b18 --- /dev/null +++ b/docs/comfyui-integration-impact.md @@ -0,0 +1,90 @@ +# Impact Assessment: Replacing Backend Generation with Configurable ComfyUI Workflows + +## 1. Executive Summary + +This document assesses the architectural impact of integrating a ComfyUI backend alongside the current local generative pipelines. The goal is to allow dynamic, runtime switching between the existing local GPU implementations and a configurable ComfyUI workflow engine. + +The integration assumes that a ComfyUI adapter already exists and that necessary generation workflows (in JSON format) are supplied with the application. The primary challenge is adapting the existing FastAPI + `AppHandler` architecture to support a modular, parameterized workflow engine without severely disrupting the strictly typed, centralized `AppState` and resource locking mechanisms. + +--- + +## 2. Current Architecture Overview + +The backend uses a local FastAPI server where endpoints delegate business logic to a centralized `AppHandler`. + +- **State Management:** A highly normalized, typed `AppState` manages limited resources (e.g., `GpuSlot`, `CpuSlot`, `DownloadingSession`). +- **Concurrency & Locking:** A single shared `RLock` protects `AppState`. Handlers follow a strict "lock -> check -> unlock -> heavy work -> lock -> update" pattern to prevent blocking the server during generation. +- **Service Boundaries:** Heavy generative tasks are isolated behind strictly typed Python Protocols in `backend/services/` (e.g., `FastVideoPipeline`, `ImageGenerationPipeline`). These protocols dictate exact Python method signatures (e.g., `generate(...)` with specific arguments). +- **Generation Lifecycle:** `GenerationHandler` tracks progress using normalized state machines (`GenerationRunning`, `GenerationComplete`, etc.). + +--- + +## 3. Proposed Architecture Options + +We must expose supplied ComfyUI workflows dynamically, allowing the UI to configure parameters (models, LoRAs) and the user to switch generation backends at runtime. + +### Option A: Adapter Service Implementations (The "Facade" Approach) + +In this approach, we retain the exact existing Service Protocols (like `FastVideoPipeline`) and introduce new implementations (e.g., `ComfyUIFastVideoPipeline`) that wrap the ComfyUI adapter and hardcode the mapping to the JSON workflows. + +* **Pros:** + * **Minimal Blast Radius:** Zero changes to `PipelinesHandler`, `GenerationHandler`, or `_routes`. + * **Maintains Strong Typing:** Preserves the existing rigid structural typing of the Python backend. +* **Cons:** + * **Hides Dynamism:** Fails the requirement of "configurable ComfyUI workflows" because adding a new workflow or exposing a new parameter (like a new LoRA node) would require modifying the Python protocol and every implementation. + * **Duplication:** We would have to maintain a 1:1 mapping of rigid Python interfaces to dynamic JSON workflows. + +### Option B: Generic Workflow Engine Service (Recommended) + +In this approach, we introduce a new generic service, `ComfyUIWorkflowEngine`, and update the `GenerationHandler` and endpoints to route requests based on the user's active configuration. + +* **Core Concept:** The backend loads JSON workflows from disk at startup (or dynamically). These workflows are parsed to identify configurable nodes (parameters, models, LoRAs). +* **Runtime Config:** A new setting in `AppSettings` (e.g., `generation_backend: Literal["local", "comfyui"]`) dictates the routing in the handlers. +* **Pros:** + * **Highly Extensible:** Adding a new workflow or parameter only requires updating the JSON file and the UI; the backend simply passes through the configuration to the ComfyUI adapter. + * **Modular:** Clearly separates local GPU logic from ComfyUI external process logic. +* **Cons:** + * Requires modifications to `AppState` to track ComfyUI jobs. + * Requires updates to request/response models (`api_types.py`) to accept dynamic key-value parameters for ComfyUI. + +--- + +## 4. Architectural Impact & Necessary Changes (Based on Option B) + +To achieve a modular integration that fulfills the dynamic configuration requirements, the following architectural changes are required. + +### 4.1. AppSettings and API Types + +* **`app_settings.py`:** Add a `generation_backend` property to allow runtime switching. +* **`api_types.py`:** Update generation request payloads (e.g., `VideoGenerationRequest`) to include an optional dictionary for dynamic workflow parameters: `workflow_params: dict[str, Any] | None = None`. +* *Note: While the backend generally avoids dynamic dicts, it is necessary here as a pass-through layer for the ComfyUI JSON schema.* + +### 4.2. Workflow Parsing and Exposure + +* **New Service:** Introduce a `WorkflowParserService` that reads the provided JSON workflows from a designated directory (e.g., `resources/workflows/`). +* **New Endpoint:** Create `GET /api/workflows` to serve the parsed parameters to the frontend, allowing the UI to dynamically render dropdowns (for models, LoRAs) and sliders based on the workflow's exposed nodes. + +### 4.3. State Management (`AppState` & `GpuSlot`) + +Currently, `GpuSlot` is strictly typed to local pipelines (e.g., `VideoPipelineState | ICLoraState | ...`). + +* **Impact:** Offloading to ComfyUI means the local GPU is *not* occupied by the FastAPI process. +* **Change:** `AppState` must be expanded. `GpuSlot` can remain for local models, but a new slot (e.g., `ExternalSlot` or `ComfyUISlot`) must be added to track the state of the ComfyUI process. +* **Locking:** The `AppHandler` lock will still protect this new slot. The lock scope remains identical: Lock -> update `ComfyUISlot` to running -> Unlock -> send HTTP request to ComfyUI -> Lock -> update progress. + +### 4.4. Generation Handler and Progress Tracking + +* **Polling vs. Callbacks:** ComfyUI typically operates asynchronously. The backend's `GenerationHandler` currently assumes it is the active runner updating progress continuously. +* **Change:** If using ComfyUI, the backend must initiate a background task (using the existing `TaskRunner`) to poll the ComfyUI adapter for progress, mapping ComfyUI's step data to the existing `GenerationProgress` state model. This ensures the frontend's `/api/generation/progress` polling endpoint remains unbroken. + +### 4.5. Model Management (Out of Scope) + +* Currently, `ModelDownloader` fetches models to a local directory. For this iteration, ComfyUI's model management is considered out of scope. ComfyUI will be responsible for locating its own models and LoRAs as specified by the parameterized JSON workflows. + +--- + +## 5. Conclusion + +To support configurable, parameterized ComfyUI workflows alongside the existing local GPU pipelines, the backend must shift slightly from its strictly typed generative protocols to a generic workflow engine. + +By introducing a generic `ComfyUIWorkflowEngine` service, expanding `AppState` to include an `ExternalSlot` for job tracking, and updating `GenerationHandler` to poll the ComfyUI adapter in the background, the application can remain modular. This approach keeps the heavy lifting in the respective processes while maintaining the centralized locking and state guarantees that the FastAPI architecture relies upon. \ No newline at end of file From 8f1302f5faa74927769ac2322b0bfa48b4f66cf5 Mon Sep 17 00:00:00 2001 From: adunato Date: Wed, 1 Apr 2026 14:44:52 +0100 Subject: [PATCH 02/12] Added reference projects folder to git ignore --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitignore b/.gitignore index 3a7b54da..4ef710dd 100644 --- a/.gitignore +++ b/.gitignore @@ -76,3 +76,5 @@ python-embed/ # Downloaded during CI for Windows NSIS installer resources/vc_redist.x64.exe + +reference_projects/ \ No newline at end of file From b57a4f26fe68e34ee3665099868bf4869ebe6959 Mon Sep 17 00:00:00 2001 From: adunato Date: Wed, 1 Apr 2026 15:02:33 +0100 Subject: [PATCH 03/12] docs: setup CR001-comfyui-integration with technique descriptions --- .../docs}/comfyui-integration-impact.md | 0 .../docs/krita-ai-diffusion-technique.md | 28 +++++++++++++++++++ .../docs/vlo-technique.md | 28 +++++++++++++++++++ 3 files changed, 56 insertions(+) rename {docs => change-requests/CR001-comfyui-integration/docs}/comfyui-integration-impact.md (100%) create mode 100644 change-requests/CR001-comfyui-integration/docs/krita-ai-diffusion-technique.md create mode 100644 change-requests/CR001-comfyui-integration/docs/vlo-technique.md diff --git a/docs/comfyui-integration-impact.md b/change-requests/CR001-comfyui-integration/docs/comfyui-integration-impact.md similarity index 100% rename from docs/comfyui-integration-impact.md rename to change-requests/CR001-comfyui-integration/docs/comfyui-integration-impact.md diff --git a/change-requests/CR001-comfyui-integration/docs/krita-ai-diffusion-technique.md b/change-requests/CR001-comfyui-integration/docs/krita-ai-diffusion-technique.md new file mode 100644 index 00000000..10a2069a --- /dev/null +++ b/change-requests/CR001-comfyui-integration/docs/krita-ai-diffusion-technique.md @@ -0,0 +1,28 @@ +# Procedural ComfyUI Node Discovery and UI Mapping in Krita AI Diffusion + +## Overview + +Krita AI Diffusion manages ComfyUI integration by abstracting the low-level node graph into a high-level Python API. This approach ensures type safety and simplifies the construction of complex workflows while remaining flexible enough to support custom nodes. + +## Node Discovery Technique + +The system relies on the `/object_info` endpoint of the ComfyUI API to procedurally discover available nodes and their respective input/output schemas. + +1. **Schema Retrieval**: Upon connection, the client fetches the full `object_info` dictionary. +2. **Type Mapping**: It maps ComfyUI types (e.g., `IMAGE`, `LATENT`, `MODEL`) to internal Python classes. +3. **Validation**: The `ComfyWorkflow` builder uses this schema to validate inputs and outputs at construction time, preventing the creation of invalid graphs before they are even sent to the server. + +## UI-to-Node Mapping + +Unlike systems that use a 1:1 visual mapping, Krita AI Diffusion uses a **Functional Mapping** approach: + +1. **Abstract Workflows**: The UI interacts with a `Workflow` class that defines high-level operations (e.g., `sampling`, `upscaling`). +2. **Builder Pattern**: These high-level operations are translated into a sequence of `ComfyWorkflow` method calls (e.g., `w.load_checkpoint()`, `w.ksampler()`). +3. **Dynamic Node IDs**: Node IDs are generated procedurally during the building process. The UI does not need to know specific node IDs; it only cares about the parameters (e.g., `seed`, `steps`, `denoise`) passed to the high-level API. +4. **Custom Workflows**: For user-provided workflows, the system imports the JSON graph and uses the `object_info` schema to identify "input" nodes (e.g., `PrimitiveNode`, `CheckpointLoaderSimple`) that should be exposed as UI widgets based on their titles or specific metadata. + +## Key Advantages + +- **Type Safety**: Prevents common "wrong input type" errors. +- **Maintainability**: Changes in the underlying ComfyUI node names can be handled by updating the mapping logic in one place. +- **User Experience**: Simplifies complex graphs into a familiar, tool-like interface for artists. diff --git a/change-requests/CR001-comfyui-integration/docs/vlo-technique.md b/change-requests/CR001-comfyui-integration/docs/vlo-technique.md new file mode 100644 index 00000000..fcfaf827 --- /dev/null +++ b/change-requests/CR001-comfyui-integration/docs/vlo-technique.md @@ -0,0 +1,28 @@ +# Procedural ComfyUI Node Discovery and UI Mapping in Vlo + +## Overview + +Vlo implements a "Proxy-Based" approach to ComfyUI node management. It allows for a highly flexible UI that can dynamically adapt to various underlying ComfyUI workflows by explicitly defining links between UI elements and backend nodes within the workflow metadata itself. + +## Node Discovery Technique + +Vlo's discovery is centered around the **Workflow Metadata** and the `properties` field of ComfyUI nodes. + +1. **Graph Introspection**: The system parses the ComfyUI JSON workflow (both API and Graph formats). +2. **Property Extraction**: It specifically looks for a custom `proxyWidgets` field within the `properties` dictionary of a node. +3. **State Synchronization**: The backend maintains a cache of the current graph state and uses it to resolve which UI-exposed parameters correspond to which internal node inputs. + +## UI-to-Node Mapping + +Vlo uses a **Proxy Widget Mapping** technique to bridge the UI and the backend: + +1. **Explicit Mapping**: The `proxyWidgets` property contains a list of mappings. Each entry defines a `target_node_id` and a `target_param_name`. +2. **Virtual Groups**: UI elements are organized into "Groups" (e.g., "Sampling Settings", "Model Selection") based on the `group_id` and `group_title` associated with the proxy mapping. +3. **Procedural UI Generation**: The frontend receives a list of these groups and their associated controls. When a user modifies a value in the UI, the backend uses the mapping to update the specific node and parameter in the graph before submission. +4. **Node Normalization**: Before sending the final graph to ComfyUI, the system "normalizes" it, ensuring that all proxy-driven changes are correctly applied to the final execution graph. + +## Key Advantages + +- **Decoupling**: The UI structure is not hardcoded to a specific node graph; it is defined *by* the graph. +- **Granular Control**: Specific parameters from different nodes can be grouped together in the UI, even if they are far apart in the actual graph. +- **Workflow Agnostic**: Any ComfyUI workflow can be "UI-enabled" simply by adding the appropriate `proxyWidgets` metadata to the nodes. From dfd7a147df91b0bce48ba2ae12333fcb9073d3c5 Mon Sep 17 00:00:00 2001 From: adunato Date: Wed, 1 Apr 2026 16:00:55 +0100 Subject: [PATCH 04/12] docs: add prerequisites and proxyWidgets findings to technique documents --- .../docs/krita-ai-diffusion-technique.md | 7 +++++++ .../docs/vlo-technique.md | 15 +++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/change-requests/CR001-comfyui-integration/docs/krita-ai-diffusion-technique.md b/change-requests/CR001-comfyui-integration/docs/krita-ai-diffusion-technique.md index 10a2069a..32c064b6 100644 --- a/change-requests/CR001-comfyui-integration/docs/krita-ai-diffusion-technique.md +++ b/change-requests/CR001-comfyui-integration/docs/krita-ai-diffusion-technique.md @@ -4,6 +4,13 @@ Krita AI Diffusion manages ComfyUI integration by abstracting the low-level node graph into a high-level Python API. This approach ensures type safety and simplifies the construction of complex workflows while remaining flexible enough to support custom nodes. +## Prerequisites + +This technique requires the installation of a **dedicated ComfyUI extension** (`ComfyUI-Krita-AI-Diffusion`). + +- **Custom Nodes**: The system relies on specific nodes (prefixed with `ETN_`, such as `ETN_LoadImageCache`, `ETN_InjectImage`, and `ETN_LoadMaskBase64`) to handle specialized tasks like in-memory image transfer and precise canvas synchronization. +- **Fixed API Contract**: The Python backend is tightly coupled to these custom nodes. Without them, the high-level builder cannot generate the necessary graph structure to communicate with Krita. + ## Node Discovery Technique The system relies on the `/object_info` endpoint of the ComfyUI API to procedurally discover available nodes and their respective input/output schemas. diff --git a/change-requests/CR001-comfyui-integration/docs/vlo-technique.md b/change-requests/CR001-comfyui-integration/docs/vlo-technique.md index fcfaf827..7f88b8bd 100644 --- a/change-requests/CR001-comfyui-integration/docs/vlo-technique.md +++ b/change-requests/CR001-comfyui-integration/docs/vlo-technique.md @@ -4,6 +4,13 @@ Vlo implements a "Proxy-Based" approach to ComfyUI node management. It allows for a highly flexible UI that can dynamically adapt to various underlying ComfyUI workflows by explicitly defining links between UI elements and backend nodes within the workflow metadata itself. +## Prerequisites + +Vlo is **workflow-agnostic** and does not require dedicated custom nodes for its core mapping functionality. + +- **Standard Nodes**: It interfaces with standard ComfyUI nodes (e.g., `LoadImage`, `CLIPTextEncode`, `KSampler`) and popular community extensions (e.g., `VideoHelperSuite`) using their default schemas. +- **Metadata-Driven**: The only requirement is the inclusion of specific metadata (`proxyWidgets`) within the JSON workflow file's `properties` field. This allows the system to remain compatible with any node as long as the mapping is explicitly defined in the graph. + ## Node Discovery Technique Vlo's discovery is centered around the **Workflow Metadata** and the `properties` field of ComfyUI nodes. @@ -26,3 +33,11 @@ Vlo uses a **Proxy Widget Mapping** technique to bridge the UI and the backend: - **Decoupling**: The UI structure is not hardcoded to a specific node graph; it is defined *by* the graph. - **Granular Control**: Specific parameters from different nodes can be grouped together in the UI, even if they are far apart in the actual graph. - **Workflow Agnostic**: Any ComfyUI workflow can be "UI-enabled" simply by adding the appropriate `proxyWidgets` metadata to the nodes. + +## Findings on `proxyWidgets` Convention + +Research indicates that `proxyWidgets` is an established **ComfyUI metadata convention** used specifically for **Subgraphs (Group Nodes)** and **Blueprints**. + +1. **Standard Purpose**: In the broader ComfyUI ecosystem, `proxyWidgets` allows a "Group Node" or "Subgraph" to expose internal parameters (like `seed`, `steps`, or `prompt`) to the top-level UI of the collapsed group. +2. **Implementation**: It is stored in the `properties` field of a node in the workflow JSON as an array of tuples: `["internal_node_id", "widget_name"]`. +3. **Vlo's Leverage**: Vlo utilizes this standard mapping logic to drive its entire dynamic UI generation. By following this convention, Vlo can interface with any complex workflow that has been organized into logical groups/subgraphs without needing custom backend code for every new node type. From cef0e0cbd467fd093ce98f3f7867e3abda48f2f6 Mon Sep 17 00:00:00 2001 From: adunato Date: Wed, 1 Apr 2026 16:05:43 +0100 Subject: [PATCH 05/12] docs: update impact assessment with architectural options --- .../docs/comfyui-integration-impact.md | 87 +++++++++---------- 1 file changed, 40 insertions(+), 47 deletions(-) diff --git a/change-requests/CR001-comfyui-integration/docs/comfyui-integration-impact.md b/change-requests/CR001-comfyui-integration/docs/comfyui-integration-impact.md index 71555b18..236ec1e2 100644 --- a/change-requests/CR001-comfyui-integration/docs/comfyui-integration-impact.md +++ b/change-requests/CR001-comfyui-integration/docs/comfyui-integration-impact.md @@ -4,7 +4,7 @@ This document assesses the architectural impact of integrating a ComfyUI backend alongside the current local generative pipelines. The goal is to allow dynamic, runtime switching between the existing local GPU implementations and a configurable ComfyUI workflow engine. -The integration assumes that a ComfyUI adapter already exists and that necessary generation workflows (in JSON format) are supplied with the application. The primary challenge is adapting the existing FastAPI + `AppHandler` architecture to support a modular, parameterized workflow engine without severely disrupting the strictly typed, centralized `AppState` and resource locking mechanisms. +Based on recent analysis of reference projects (Krita AI Diffusion and Vlo), this document focuses on clarifying the architectural options for procedural node discovery and UI mapping. The objective of this phase is to review these options prior to committing to a final architectural decision. --- @@ -14,77 +14,70 @@ The backend uses a local FastAPI server where endpoints delegate business logic - **State Management:** A highly normalized, typed `AppState` manages limited resources (e.g., `GpuSlot`, `CpuSlot`, `DownloadingSession`). - **Concurrency & Locking:** A single shared `RLock` protects `AppState`. Handlers follow a strict "lock -> check -> unlock -> heavy work -> lock -> update" pattern to prevent blocking the server during generation. -- **Service Boundaries:** Heavy generative tasks are isolated behind strictly typed Python Protocols in `backend/services/` (e.g., `FastVideoPipeline`, `ImageGenerationPipeline`). These protocols dictate exact Python method signatures (e.g., `generate(...)` with specific arguments). +- **Service Boundaries:** Heavy generative tasks are isolated behind strictly typed Python Protocols in `backend/services/` (e.g., `FastVideoPipeline`). - **Generation Lifecycle:** `GenerationHandler` tracks progress using normalized state machines (`GenerationRunning`, `GenerationComplete`, etc.). --- -## 3. Proposed Architecture Options +## 3. Proposed Architecture Options for UI-to-Node Mapping -We must expose supplied ComfyUI workflows dynamically, allowing the UI to configure parameters (models, LoRAs) and the user to switch generation backends at runtime. +A core challenge is how the frontend UI (sliders, dropdowns for models/LoRAs) dynamically maps to and controls the underlying ComfyUI node graph. Two distinct architectural options have been identified based on industry reference projects. -### Option A: Adapter Service Implementations (The "Facade" Approach) +### Option A: Functional Mapping via Dedicated Custom Nodes (The Krita Approach) -In this approach, we retain the exact existing Service Protocols (like `FastVideoPipeline`) and introduce new implementations (e.g., `ComfyUIFastVideoPipeline`) that wrap the ComfyUI adapter and hardcode the mapping to the JSON workflows. +In this approach, the integration relies on abstracting the low-level node graph into a high-level Python API, tightly coupled with a dedicated suite of custom ComfyUI nodes. +* **Mechanism:** + * Relies on the ComfyUI `/object_info` API to discover available nodes and their schemas. + * The backend implements a "Builder Pattern" (`ComfyWorkflow`) that translates high-level UI requests (e.g., `generate_video(prompt, seed)`) into specific node instantiations. + * **Crucial Prerequisite:** Requires installing a dedicated ComfyUI extension with custom nodes (e.g., `LTX_LoadImage`, `LTX_InjectVideo`) designed specifically to bridge the communication gap (e.g., handling in-memory transfers or specific app logic). * **Pros:** - * **Minimal Blast Radius:** Zero changes to `PipelinesHandler`, `GenerationHandler`, or `_routes`. - * **Maintains Strong Typing:** Preserves the existing rigid structural typing of the Python backend. + * **Strong Type Safety:** The builder validates inputs/outputs against the `object_info` schema before execution. + * **Simpler UI Logic:** The UI only interacts with high-level parameters; the backend handles the complex graph construction. * **Cons:** - * **Hides Dynamism:** Fails the requirement of "configurable ComfyUI workflows" because adding a new workflow or exposing a new parameter (like a new LoRA node) would require modifying the Python protocol and every implementation. - * **Duplication:** We would have to maintain a 1:1 mapping of rigid Python interfaces to dynamic JSON workflows. + * **High Maintenance:** Tightly coupled to specific custom nodes. Any changes to the node logic require updating the backend builder. + * **Less Flexible:** Harder for users to drop in arbitrary, wildly different ComfyUI workflows without backend updates. -### Option B: Generic Workflow Engine Service (Recommended) +### Option B: Proxy-Based Metadata Mapping (The Vlo Approach) -In this approach, we introduce a new generic service, `ComfyUIWorkflowEngine`, and update the `GenerationHandler` and endpoints to route requests based on the user's active configuration. +This approach is workflow-agnostic and relies on embedding UI mapping rules directly within the ComfyUI workflow JSON metadata. -* **Core Concept:** The backend loads JSON workflows from disk at startup (or dynamically). These workflows are parsed to identify configurable nodes (parameters, models, LoRAs). -* **Runtime Config:** A new setting in `AppSettings` (e.g., `generation_backend: Literal["local", "comfyui"]`) dictates the routing in the handlers. +* **Mechanism:** + * Relies on a custom metadata field, specifically the established ComfyUI convention `proxyWidgets`, located within the `properties` dictionary of Subgraphs (Group Nodes) or individual nodes. + * The workflow JSON explicitly defines mapping tuples (e.g., `["node_id_31", "seed"]`). + * The backend parses these JSON files at startup, discovers the exposed parameters, and serves this dynamic schema to the frontend. + * **Crucial Prerequisite:** Requires zero dedicated custom nodes. It interfaces with standard ComfyUI nodes and relies entirely on the JSON metadata structure. * **Pros:** - * **Highly Extensible:** Adding a new workflow or parameter only requires updating the JSON file and the UI; the backend simply passes through the configuration to the ComfyUI adapter. - * **Modular:** Clearly separates local GPU logic from ComfyUI external process logic. + * **Highly Decoupled & Workflow Agnostic:** The UI structure is defined *by* the graph. Users can drop in entirely new workflows (using standard nodes) as long as they tag the inputs with `proxyWidgets`. + * **Minimal Backend Logic:** The backend acts primarily as a pass-through and normalizer, rather than a complex graph builder. * **Cons:** - * Requires modifications to `AppState` to track ComfyUI jobs. - * Requires updates to request/response models (`api_types.py`) to accept dynamic key-value parameters for ComfyUI. + * **Weaker Typing:** Relies on dynamic dictionaries passing through the backend, requiring careful validation logic. + * **Complex JSON Maintenance:** The burden of defining the UI shifts to whoever authors the ComfyUI JSON workflows; they must correctly set up the `proxyWidgets` arrays. --- -## 4. Architectural Impact & Necessary Changes (Based on Option B) +## 4. Architectural Impact on LTX-Desktop Backend -To achieve a modular integration that fulfills the dynamic configuration requirements, the following architectural changes are required. +Regardless of the option chosen, migrating to a ComfyUI engine impacts the core architecture: ### 4.1. AppSettings and API Types +* **Runtime Config:** `app_settings.py` needs a `generation_backend` flag. +* **Dynamic Payloads:** `api_types.py` must be updated. Option B requires highly dynamic `workflow_params: dict[str, Any]` to accommodate arbitrary proxy widgets, whereas Option A might allow slightly more structured, high-level requests. -* **`app_settings.py`:** Add a `generation_backend` property to allow runtime switching. -* **`api_types.py`:** Update generation request payloads (e.g., `VideoGenerationRequest`) to include an optional dictionary for dynamic workflow parameters: `workflow_params: dict[str, Any] | None = None`. -* *Note: While the backend generally avoids dynamic dicts, it is necessary here as a pass-through layer for the ComfyUI JSON schema.* +### 4.2. State Management (`AppState` & `GpuSlot`) +* Currently, `GpuSlot` is strictly typed to local pipelines. +* A new slot (e.g., `ComfyUISlot`) must be added to track the state of the external ComfyUI process. The `AppHandler` lock will protect this slot similarly to local execution. -### 4.2. Workflow Parsing and Exposure - -* **New Service:** Introduce a `WorkflowParserService` that reads the provided JSON workflows from a designated directory (e.g., `resources/workflows/`). -* **New Endpoint:** Create `GET /api/workflows` to serve the parsed parameters to the frontend, allowing the UI to dynamically render dropdowns (for models, LoRAs) and sliders based on the workflow's exposed nodes. - -### 4.3. State Management (`AppState` & `GpuSlot`) - -Currently, `GpuSlot` is strictly typed to local pipelines (e.g., `VideoPipelineState | ICLoraState | ...`). - -* **Impact:** Offloading to ComfyUI means the local GPU is *not* occupied by the FastAPI process. -* **Change:** `AppState` must be expanded. `GpuSlot` can remain for local models, but a new slot (e.g., `ExternalSlot` or `ComfyUISlot`) must be added to track the state of the ComfyUI process. -* **Locking:** The `AppHandler` lock will still protect this new slot. The lock scope remains identical: Lock -> update `ComfyUISlot` to running -> Unlock -> send HTTP request to ComfyUI -> Lock -> update progress. - -### 4.4. Generation Handler and Progress Tracking - -* **Polling vs. Callbacks:** ComfyUI typically operates asynchronously. The backend's `GenerationHandler` currently assumes it is the active runner updating progress continuously. -* **Change:** If using ComfyUI, the backend must initiate a background task (using the existing `TaskRunner`) to poll the ComfyUI adapter for progress, mapping ComfyUI's step data to the existing `GenerationProgress` state model. This ensures the frontend's `/api/generation/progress` polling endpoint remains unbroken. - -### 4.5. Model Management (Out of Scope) - -* Currently, `ModelDownloader` fetches models to a local directory. For this iteration, ComfyUI's model management is considered out of scope. ComfyUI will be responsible for locating its own models and LoRAs as specified by the parameterized JSON workflows. +### 4.3. Generation Handler and Progress Tracking +* ComfyUI operates asynchronously. The backend must introduce a background polling mechanism (via `TaskRunner`) or websocket listener to track progress and map ComfyUI's step data to the existing `GenerationProgress` state model, keeping the frontend polling endpoint intact. --- -## 5. Conclusion +## 5. Conclusion and Next Steps + +The decision between **Option A (Functional/Custom Nodes)** and **Option B (Proxy Metadata/Agnostic)** dictates the entire trajectory of the backend integration. -To support configurable, parameterized ComfyUI workflows alongside the existing local GPU pipelines, the backend must shift slightly from its strictly typed generative protocols to a generic workflow engine. +* Option A favors strict control, type safety, and specialized custom node logic but sacrifices user workflow flexibility. +* Option B favors extreme flexibility and relies on established ComfyUI UI conventions (`proxyWidgets`), pushing the configuration burden to the workflow JSON authors. -By introducing a generic `ComfyUIWorkflowEngine` service, expanding `AppState` to include an `ExternalSlot` for job tracking, and updating `GenerationHandler` to poll the ComfyUI adapter in the background, the application can remain modular. This approach keeps the heavy lifting in the respective processes while maintaining the centralized locking and state guarantees that the FastAPI architecture relies upon. \ No newline at end of file +**Next Step:** This document serves as the basis for an architectural review. A final decision on Option A vs. Option B must be made before implementation of the ComfyUI adapter service begins. \ No newline at end of file From 86a86ef3baa91417aa003d6263e098863ec383bd Mon Sep 17 00:00:00 2001 From: adunato Date: Wed, 1 Apr 2026 16:10:48 +0100 Subject: [PATCH 06/12] docs: finalize architecture choice selecting Option B --- .../docs/comfyui-integration-impact.md | 43 ++++++++++++------- 1 file changed, 27 insertions(+), 16 deletions(-) diff --git a/change-requests/CR001-comfyui-integration/docs/comfyui-integration-impact.md b/change-requests/CR001-comfyui-integration/docs/comfyui-integration-impact.md index 236ec1e2..3221d984 100644 --- a/change-requests/CR001-comfyui-integration/docs/comfyui-integration-impact.md +++ b/change-requests/CR001-comfyui-integration/docs/comfyui-integration-impact.md @@ -56,28 +56,39 @@ This approach is workflow-agnostic and relies on embedding UI mapping rules dire --- -## 4. Architectural Impact on LTX-Desktop Backend +## 4. Architectural Impact on LTX-Desktop Backend (Option B Selected) -Regardless of the option chosen, migrating to a ComfyUI engine impacts the core architecture: +Following review, **Option B (Proxy-Based Metadata Mapping)** has been selected. The primary directive for this integration is to **maximize out-of-the-box compatibility** while ensuring a **minimal fork update impact**. -### 4.1. AppSettings and API Types -* **Runtime Config:** `app_settings.py` needs a `generation_backend` flag. -* **Dynamic Payloads:** `api_types.py` must be updated. Option B requires highly dynamic `workflow_params: dict[str, Any]` to accommodate arbitrary proxy widgets, whereas Option A might allow slightly more structured, high-level requests. +Crucially, the ComfyUI integration must be added *on top* of existing services. Current LTX Desktop functionality (local, native GPU generation) must be fully retained and operate exactly as before when ComfyUI is not active. By isolating the new ComfyUI logic, we ensure that upstream merges from the original `LTX-Desktop` repository remain trivial. -### 4.2. State Management (`AppState` & `GpuSlot`) -* Currently, `GpuSlot` is strictly typed to local pipelines. -* A new slot (e.g., `ComfyUISlot`) must be added to track the state of the external ComfyUI process. The `AppHandler` lock will protect this slot similarly to local execution. +### 4.1. Core Principle: Isolation for Minimal Merge Conflicts +To minimize merge conflicts when pulling from the upstream fork, the ComfyUI integration will avoid heavily modifying existing core files (like `app_handler.py` or complex state machines) wherever possible. Instead, it will rely on new interface implementations and isolated modules. -### 4.3. Generation Handler and Progress Tracking -* ComfyUI operates asynchronously. The backend must introduce a background polling mechanism (via `TaskRunner`) or websocket listener to track progress and map ComfyUI's step data to the existing `GenerationProgress` state model, keeping the frontend polling endpoint intact. +* **New Modules:** All ComfyUI-specific parsing, proxy mapping, and network communication will live in a strictly separated directory (e.g., `backend/services/comfyui/`). +* **Interface Implementation:** The ComfyUI engine will implement the existing pipeline interfaces (e.g., creating a `ComfyUIVideoPipeline` that adheres to the same Protocol as `FastVideoPipeline`). This allows the core `GenerationHandler` to treat it as just another backend without knowing its internal complexities. ---- +### 4.2. AppSettings and API Types (Additive Changes) +Changes to core API files will be strictly additive, preserving all existing schemas. +* **`app_settings.py`:** Add a new, optional `generation_backend` flag (defaulting to `local`). +* **`api_types.py`:** Add a new, optional `workflow_params: dict[str, Any] | None = None` to the generation request models to handle the dynamic `proxyWidgets` inputs. Existing strictly-typed fields (prompt, seed, etc.) remain untouched and can be mapped internally if ComfyUI is the active backend. + +### 4.3. State Management (`AppState` & `GpuSlot`) +The existing `GpuSlot` logic, which carefully manages local VRAM, must remain untouched to ensure the default local generation experience is not compromised. +* **Additive State:** A new, distinct slot (e.g., `ExternalSlot` or `ComfyUIJobSlot`) will be introduced to `AppState`. +* **Locking:** The `AppHandler` will use the same locking mechanism to protect this new slot, ensuring thread safety without needing to rewrite the existing local GPU locking logic. If the active backend is ComfyUI, the handler checks the `ExternalSlot` instead of the `GpuSlot`. -## 5. Conclusion and Next Steps +### 4.4. Generation Handler and Progress Tracking +The existing `GenerationHandler` relies heavily on the local process actively reporting progress. +* **Adapter Pattern:** A `ComfyUIAdapter` service will act as a bridge. When a generation task is dispatched to ComfyUI, the adapter will use the `TaskRunner` to spawn a background polling task. +* **State Translation:** This polling task will fetch ComfyUI's native progress (using its API/Websocket) and translate it into the *exact same* `GenerationProgress` state objects currently expected by the frontend. This ensures the frontend UI requires minimal, if any, modifications to display progress bars. + +--- -The decision between **Option A (Functional/Custom Nodes)** and **Option B (Proxy Metadata/Agnostic)** dictates the entire trajectory of the backend integration. +## 5. Conclusion -* Option A favors strict control, type safety, and specialized custom node logic but sacrifices user workflow flexibility. -* Option B favors extreme flexibility and relies on established ComfyUI UI conventions (`proxyWidgets`), pushing the configuration burden to the workflow JSON authors. +By selecting **Option B (Proxy-Based Metadata Mapping)**, we achieve a highly flexible, workflow-agnostic system. -**Next Step:** This document serves as the basis for an architectural review. A final decision on Option A vs. Option B must be made before implementation of the ComfyUI adapter service begins. \ No newline at end of file +By applying a strict **"Additive Isolation"** architectural lens, we ensure that: +1. **Original functionality is preserved:** Local generation remains completely unaffected and acts as the default. +2. **Upstream merges are trivial:** By avoiding modifications to core logic loops and instead adding new interface implementations and isolated service folders, we minimize the "fork divergence," allowing the project to easily consume future updates from the original Lightricks repository. \ No newline at end of file From 053dd680fac97ccf9d8c4c54a16d625b64dfd430 Mon Sep 17 00:00:00 2001 From: adunato Date: Wed, 1 Apr 2026 16:15:39 +0100 Subject: [PATCH 07/12] docs: create HLD and Implementation Plan for ComfyUI integration --- .../CR001-comfyui-integration/HLD.md | 57 ++++++++++++++ .../IMPLEMENTATION_PLAN.md | 76 +++++++++++++++++++ 2 files changed, 133 insertions(+) create mode 100644 change-requests/CR001-comfyui-integration/HLD.md create mode 100644 change-requests/CR001-comfyui-integration/IMPLEMENTATION_PLAN.md diff --git a/change-requests/CR001-comfyui-integration/HLD.md b/change-requests/CR001-comfyui-integration/HLD.md new file mode 100644 index 00000000..f77dabe9 --- /dev/null +++ b/change-requests/CR001-comfyui-integration/HLD.md @@ -0,0 +1,57 @@ +# High-Level Design (HLD): ComfyUI Integration + +## 1. Overview + +This High-Level Design details the integration of a ComfyUI engine into the LTX-Desktop application. As determined in the Impact Assessment, this integration follows **Option B: Proxy-Based Metadata Mapping**. + +The core architectural philosophy is **Additive Isolation**: the ComfyUI integration will be built *on top* of the existing local generation capabilities without modifying their underlying logic. This minimizes merge conflicts with the upstream fork and ensures the default native GPU experience is preserved. + +## 2. Core Components + +### 2.1. App Settings and API Types (Additive) + +* **`AppSettings`**: A new setting, `generation_backend` (Literal: `"local" | "comfyui"`), will be added to dictate the routing logic. +* **`api_types.py`**: Generation request payloads (e.g., `VideoGenerationRequest`) will be extended with an optional `workflow_params: dict[str, Any] | None` to pass dynamic proxy widget values from the UI to the backend. + +### 2.2. State Management (`AppState`) + +To avoid disrupting the highly tuned local `GpuSlot` management: +* A new state slot, **`ComfyUIJobSlot`**, will be introduced in `AppState`. +* The centralized `AppHandler` lock will protect this new slot exactly as it protects the `GpuSlot`. + +### 2.3. ComfyUI Service Module + +A new, isolated module (`backend/services/comfyui/`) will encapsulate all ComfyUI-specific logic: + +1. **`WorkflowParser`**: + * Reads predefined ComfyUI JSON workflows. + * Extracts the `proxyWidgets` metadata to identify which internal node parameters are exposed to the UI. +2. **`ComfyUIClient`**: + * Handles HTTP communication with the ComfyUI server (e.g., `/prompt`, `/upload/image`, `/history`). + * Manages WebSocket connections (if required) for real-time progress updates. +3. **`ComfyUIPipelineAdapters`**: + * Implements the existing strictly-typed protocols (e.g., `FastVideoPipeline`). + * Translates the incoming `VideoGenerationRequest` (including `workflow_params`) into the final execution graph JSON. + +### 2.4. Generation Handler Routing + +The `GenerationHandler` will act as a router based on the `generation_backend` setting: + +* **If `"local"`**: The handler proceeds normally, acquiring the `GpuSlot` and delegating to the native `services.video_processor`. +* **If `"comfyui"`**: The handler bypasses the `GpuSlot`, acquires the `ComfyUIJobSlot`, and delegates to the `ComfyUIPipelineAdapter`. + +### 2.5. Progress Translation + +To ensure the frontend requires zero changes to its progress tracking logic: +* The `ComfyUIPipelineAdapter` will spawn a background polling task (using the existing `TaskRunner`). +* This task will translate ComfyUI's native execution progress into the exact `GenerationProgress` (e.g., `GenerationRunning`, `GenerationComplete`) state objects expected by `AppState`. + +## 3. Architectural Flow (ComfyUI Active) + +1. **UI Configuration**: Frontend fetches available workflows via a new endpoint (parsed by `WorkflowParser`) and dynamically renders controls for the exposed `proxyWidgets`. +2. **Submission**: User clicks generate. Frontend sends `VideoGenerationRequest` including `workflow_params`. +3. **Routing**: `GenerationHandler` sees `generation_backend == "comfyui"`. +4. **Locking**: Handler acquires lock -> sets `ComfyUIJobSlot` to running -> unlocks. +5. **Execution**: `ComfyUIPipelineAdapter` constructs the final JSON graph and sends it to the `ComfyUIClient`. +6. **Progress**: Background task polls ComfyUI, locking briefly to update `ComfyUIJobSlot` progress. +7. **Completion**: Adapter retrieves the final media from ComfyUI, saves it locally, and updates state to `GenerationComplete`. diff --git a/change-requests/CR001-comfyui-integration/IMPLEMENTATION_PLAN.md b/change-requests/CR001-comfyui-integration/IMPLEMENTATION_PLAN.md new file mode 100644 index 00000000..11e0be68 --- /dev/null +++ b/change-requests/CR001-comfyui-integration/IMPLEMENTATION_PLAN.md @@ -0,0 +1,76 @@ +# Implementation Plan: ComfyUI Integration + +This plan outlines the steps required to implement the ComfyUI integration described in the High-Level Design (HLD), adhering to the principles of "Additive Isolation" to minimize fork impact. + +## Phase 1: Foundation (Settings & API Types) + +**Goal:** Extend the core data structures to support routing and dynamic payload parameters without breaking existing schemas. + +1. **Update Settings:** + * Modify `settings.json` and `backend/state/app_settings.py` to add `generation_backend` (defaulting to `"local"`). +2. **Update API Types:** + * In `backend/api_types.py`, add `workflow_params: dict[str, Any] | None = None` to generation requests (e.g., `VideoGenerationRequest`). +3. **Update State Types:** + * In `backend/state/app_state_types.py`, define a new `ComfyUIJobSlot` (tracking status, progress, current job ID). + * Add `comfyui_job: ComfyUIJobSlot | None` to the root `AppState` definition. + +## Phase 2: Core ComfyUI Services + +**Goal:** Create the isolated module (`backend/services/comfyui/`) for parsing workflows and communicating with the ComfyUI server. + +1. **Create Service Directory:** Initialize `backend/services/comfyui/`. +2. **Implement `WorkflowParser`:** + * Create logic to read JSON workflows from a designated directory. + * Extract `proxyWidgets` metadata from node properties. +3. **Implement `ComfyUIClient`:** + * Create an asynchronous HTTP client to communicate with the ComfyUI API (`/prompt`, `/upload/image`, `/history`, `/view`). +4. **Expose Workflows Endpoint:** + * Create a new route in `backend/_routes/` (e.g., `workflows.py`) to expose the parsed workflows and their configurable parameters to the frontend. + * Wire the route into `app_factory.py`. + +## Phase 3: Pipeline Adapters and Progress Tracking + +**Goal:** Implement the adapter that bridges the strictly typed LTX-Desktop protocols with the dynamic ComfyUI engine. + +1. **Implement `ComfyUIVideoPipeline`:** + * Create a class implementing the `FastVideoPipeline` (or relevant) protocol. + * Implement graph construction: merge the base JSON workflow with the incoming `workflow_params`. +2. **Implement Background Polling:** + * Use the existing `TaskRunner` to spawn a background task upon job submission. + * Poll the ComfyUI server for progress on the submitted `prompt_id`. + * Translate the progress into standard `GenerationProgress` state objects. + +## Phase 4: Handler Routing & Locking Integration + +**Goal:** Update the centralized handler to securely route tasks based on the active backend. + +1. **Update `GenerationHandler`:** + * In `backend/handlers/generation_handler.py`, read `generation_backend` from settings. + * Implement branching logic: + * If `"local"`: Use `GpuSlot` and standard pipeline logic (existing code). + * If `"comfyui"`: Acquire lock, validate `ComfyUIJobSlot` is idle, set to running, release lock, and dispatch to `ComfyUIVideoPipeline`. +2. **Ensure Lock Safety:** + * Verify the "lock -> check -> unlock -> heavy work -> lock -> update" pattern is strictly followed for the new `ComfyUIJobSlot`. + +## Phase 5: Frontend Integration + +**Goal:** Update the React frontend to dynamically render UI elements based on the parsed ComfyUI workflows. + +1. **Backend Toggle:** Add a UI toggle in the settings to switch between Local and ComfyUI backends. +2. **Fetch Workflows:** On mount (if ComfyUI is active), fetch the available workflows from the new backend endpoint. +3. **Dynamic Rendering:** + * Parse the returned proxy widget schemas. + * Dynamically render sliders, dropdowns, and text inputs based on the expected types of the proxy widgets. +4. **Submission Logic:** Update the `backendFetch` calls for generation to include the user-configured `workflow_params` dictionary. + +## Phase 6: Testing and Validation + +**Goal:** Ensure the integration is robust and the local pipeline remains unaffected. + +1. **Backend Integration Tests:** + * Create new tests in `backend/tests/` using fakes for the `ComfyUIClient`. + * Verify routing logic works correctly based on settings. +2. **Type Checking:** + * Run `pnpm typecheck` to ensure the new dynamic dictionaries haven't violated strict mode rules elsewhere. +3. **Local Regression:** + * Run existing `backend:test` suite to guarantee standard local generation is completely isolated and functional. \ No newline at end of file From f9982ca992819ecaba96b01d4167bec88cafbd8d Mon Sep 17 00:00:00 2001 From: adunato Date: Wed, 1 Apr 2026 16:22:27 +0100 Subject: [PATCH 08/12] build: add pnpm dev to VS Code launch configurations --- .vscode/launch.json | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/.vscode/launch.json b/.vscode/launch.json index 84ae6ae4..8e55d5d8 100644 --- a/.vscode/launch.json +++ b/.vscode/launch.json @@ -1,6 +1,17 @@ { "version": "0.2.0", "configurations": [ + { + "name": "Run: pnpm dev", + "type": "node", + "request": "launch", + "runtimeExecutable": "pnpm", + "runtimeArgs": [ + "dev" + ], + "cwd": "${workspaceFolder}", + "console": "integratedTerminal" + }, { "name": "Attach: Electron Main", "type": "node", From ad7078b57cdd608bb7bab07fffde602a15a00f6d Mon Sep 17 00:00:00 2001 From: adunato Date: Wed, 1 Apr 2026 16:32:05 +0100 Subject: [PATCH 09/12] feat: allow bypass of API key check via LTX_BYPASS_API_CHECK for local dev --- backend/runtime_config/runtime_policy.py | 4 ++++ package.json | 4 ++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/backend/runtime_config/runtime_policy.py b/backend/runtime_config/runtime_policy.py index 7b84dc89..40f9312f 100644 --- a/backend/runtime_config/runtime_policy.py +++ b/backend/runtime_config/runtime_policy.py @@ -1,10 +1,14 @@ """Runtime policy decisions for forced API mode.""" from __future__ import annotations +import os def decide_force_api_generations(system: str, cuda_available: bool, vram_gb: int | None) -> bool: """Return whether API-only generation must be forced for this runtime.""" + if os.environ.get("LTX_BYPASS_API_CHECK") == "1": + return False + if system == "Darwin": return True diff --git a/package.json b/package.json index 2204df1a..3aa8f223 100644 --- a/package.json +++ b/package.json @@ -14,8 +14,8 @@ "packageManager": "pnpm@10.30.3", "scripts": { "setup:dev": "node scripts/run-script.js scripts/setup-dev", - "dev": "vite", - "dev:debug": "cross-env BACKEND_DEBUG=1 ELECTRON_DEBUG=1 vite", + "dev": "cross-env LTX_BYPASS_API_CHECK=1 vite", + "dev:debug": "cross-env BACKEND_DEBUG=1 ELECTRON_DEBUG=1 LTX_BYPASS_API_CHECK=1 vite", "typecheck": "concurrently \"pnpm run typecheck:ts\" \"pnpm run typecheck:py\"", "typecheck:ts": "tsc --noEmit", "typecheck:py": "cd backend && uv run pyright", From af2c2cb2c90eff477b59316b92c2f774a5007f4d Mon Sep 17 00:00:00 2001 From: adunato Date: Wed, 1 Apr 2026 18:56:53 +0100 Subject: [PATCH 10/12] docs: add HLD for CR002-developer-configuration --- .../CR002-developer-configuration/HLD.md | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 change-requests/CR002-developer-configuration/HLD.md diff --git a/change-requests/CR002-developer-configuration/HLD.md b/change-requests/CR002-developer-configuration/HLD.md new file mode 100644 index 00000000..b485bf8b --- /dev/null +++ b/change-requests/CR002-developer-configuration/HLD.md @@ -0,0 +1,42 @@ +# High-Level Design (HLD): Developer Configuration - API Check Bypass + +## 1. Overview + +This document outlines the changes made for **CR002: Developer Configuration** to streamline the local development experience. The primary goal of this iteration is to allow developers to easily bypass the mandatory "Connect API Keys" gate that blocks access to the application UI on machines lacking sufficient hardware or proper setup. + +The solution ensures that developers can access and work on the application UI (e.g., for integrating ComfyUI workflows) without needing actual API keys or a production-ready local GPU environment, while strictly maintaining the integrity of the production codebase. + +## 2. Core Components + +### 2.1. Backend Runtime Policy + +The single source of truth for whether the application requires API keys is the `decide_force_api_generations` function located in `backend/runtime_config/runtime_policy.py`. + +* **Change:** An environment variable check (`LTX_BYPASS_API_CHECK`) was introduced into this function. +* **Logic:** If `os.environ.get("LTX_BYPASS_API_CHECK") == "1"`, the function immediately returns `False`. +* **Impact:** Returning `False` forces the backend to report that local generation is allowed, which in turn signals the frontend to dismiss the blocking API Key modal. + +### 2.2. Development Scripts configuration + +To ensure this bypass is seamless and strictly limited to local development, the environment variable is injected via `package.json` scripts rather than relying on a global `.env` file. + +* **Change:** The `dev` and `dev:debug` scripts in `package.json` were updated using `cross-env`. +* **Old Scripts:** + ```json + "dev": "vite", + "dev:debug": "cross-env BACKEND_DEBUG=1 ELECTRON_DEBUG=1 vite" + ``` +* **New Scripts:** + ```json + "dev": "cross-env LTX_BYPASS_API_CHECK=1 vite", + "dev:debug": "cross-env BACKEND_DEBUG=1 ELECTRON_DEBUG=1 LTX_BYPASS_API_CHECK=1 vite" + ``` + +## 3. Security and Production Safety + +* **No Global `.env`:** By avoiding a `.env` file, we prevent accidental commits or persistent global state that could inadvertently disable the API check outside of an active `pnpm dev` session. +* **Production Build Isolation:** The `build`, `build:skip-python`, and `build:fast` scripts in `package.json` do **not** include the `LTX_BYPASS_API_CHECK` flag. Consequently, packaged release binaries remain secure and will correctly enforce hardware checks and API key requirements for end-users. + +## 4. Conclusion + +This non-invasive approach provides a frictionless path for frontend and integration development without compromising the strict hardware and licensing gates required for the application's production release. \ No newline at end of file From 67483187b257f91b945c2252a4c5f162fc8e9017 Mon Sep 17 00:00:00 2001 From: adunato Date: Wed, 1 Apr 2026 19:03:19 +0100 Subject: [PATCH 11/12] feat: bypass model check via LTX_BYPASS_MODEL_CHECK for local dev --- backend/ltx2_server.py | 6 ++++- .../CR002-developer-configuration/HLD.md | 27 ++++++++++++------- package.json | 4 +-- 3 files changed, 24 insertions(+), 13 deletions(-) diff --git a/backend/ltx2_server.py b/backend/ltx2_server.py index c7898f5f..78d1b9e8 100644 --- a/backend/ltx2_server.py +++ b/backend/ltx2_server.py @@ -181,9 +181,13 @@ def _resolve_force_api_generations() -> bool: return force_api_generations +import os + FORCE_API_GENERATIONS = _resolve_force_api_generations() +_BYPASS_MODEL_CHECK = os.environ.get("LTX_BYPASS_MODEL_CHECK") == "1" + REQUIRED_MODEL_TYPES: frozenset[ModelFileType] = ( - frozenset() if FORCE_API_GENERATIONS else DEFAULT_REQUIRED_MODEL_TYPES + frozenset() if (FORCE_API_GENERATIONS or _BYPASS_MODEL_CHECK) else DEFAULT_REQUIRED_MODEL_TYPES ) CAMERA_MOTION_PROMPTS = { diff --git a/change-requests/CR002-developer-configuration/HLD.md b/change-requests/CR002-developer-configuration/HLD.md index b485bf8b..1f9df583 100644 --- a/change-requests/CR002-developer-configuration/HLD.md +++ b/change-requests/CR002-developer-configuration/HLD.md @@ -8,17 +8,24 @@ The solution ensures that developers can access and work on the application UI ( ## 2. Core Components -### 2.1. Backend Runtime Policy +### 2.1. Backend Runtime Policy & Model Checking -The single source of truth for whether the application requires API keys is the `decide_force_api_generations` function located in `backend/runtime_config/runtime_policy.py`. +The application normally enforces two major gates before allowing access to the main UI: +1. **API Key Gate:** Triggered if the system lacks the necessary hardware to run local models (managed by `decide_force_api_generations`). +2. **Model Download Gate:** Triggered if local generation is allowed but the required model files (e.g., checkpoints) are missing on disk. -* **Change:** An environment variable check (`LTX_BYPASS_API_CHECK`) was introduced into this function. -* **Logic:** If `os.environ.get("LTX_BYPASS_API_CHECK") == "1"`, the function immediately returns `False`. -* **Impact:** Returning `False` forces the backend to report that local generation is allowed, which in turn signals the frontend to dismiss the blocking API Key modal. +To bypass these gates, two environment variables were introduced: + +* **`LTX_BYPASS_API_CHECK`:** + * **Location:** `backend/runtime_config/runtime_policy.py` (`decide_force_api_generations`) + * **Logic:** If set to `"1"`, the function returns `False`, tricking the app into believing local generation is fully supported, thereby dismissing the mandatory API key prompt. +* **`LTX_BYPASS_MODEL_CHECK`:** + * **Location:** `backend/ltx2_server.py` + * **Logic:** If set to `"1"`, the `REQUIRED_MODEL_TYPES` frozen set is overridden to be empty (`frozenset()`). This tricks the frontend's model status check into believing all necessary models are already downloaded, skipping the "Choose Location" installation screen. ### 2.2. Development Scripts configuration -To ensure this bypass is seamless and strictly limited to local development, the environment variable is injected via `package.json` scripts rather than relying on a global `.env` file. +To ensure these bypasses are seamless and strictly limited to local development, the environment variables are injected via `package.json` scripts rather than relying on a global `.env` file. * **Change:** The `dev` and `dev:debug` scripts in `package.json` were updated using `cross-env`. * **Old Scripts:** @@ -28,14 +35,14 @@ To ensure this bypass is seamless and strictly limited to local development, the ``` * **New Scripts:** ```json - "dev": "cross-env LTX_BYPASS_API_CHECK=1 vite", - "dev:debug": "cross-env BACKEND_DEBUG=1 ELECTRON_DEBUG=1 LTX_BYPASS_API_CHECK=1 vite" + "dev": "cross-env LTX_BYPASS_API_CHECK=1 LTX_BYPASS_MODEL_CHECK=1 vite", + "dev:debug": "cross-env BACKEND_DEBUG=1 ELECTRON_DEBUG=1 LTX_BYPASS_API_CHECK=1 LTX_BYPASS_MODEL_CHECK=1 vite" ``` ## 3. Security and Production Safety -* **No Global `.env`:** By avoiding a `.env` file, we prevent accidental commits or persistent global state that could inadvertently disable the API check outside of an active `pnpm dev` session. -* **Production Build Isolation:** The `build`, `build:skip-python`, and `build:fast` scripts in `package.json` do **not** include the `LTX_BYPASS_API_CHECK` flag. Consequently, packaged release binaries remain secure and will correctly enforce hardware checks and API key requirements for end-users. +* **No Global `.env`:** By avoiding a `.env` file, we prevent accidental commits or persistent global state that could inadvertently disable the API or model checks outside of an active `pnpm dev` session. +* **Production Build Isolation:** The `build`, `build:skip-python`, and `build:fast` scripts in `package.json` do **not** include these bypass flags. Consequently, packaged release binaries remain secure and will correctly enforce hardware checks, API key requirements, and model downloads for end-users. ## 4. Conclusion diff --git a/package.json b/package.json index 3aa8f223..be06b5a0 100644 --- a/package.json +++ b/package.json @@ -14,8 +14,8 @@ "packageManager": "pnpm@10.30.3", "scripts": { "setup:dev": "node scripts/run-script.js scripts/setup-dev", - "dev": "cross-env LTX_BYPASS_API_CHECK=1 vite", - "dev:debug": "cross-env BACKEND_DEBUG=1 ELECTRON_DEBUG=1 LTX_BYPASS_API_CHECK=1 vite", + "dev": "cross-env LTX_BYPASS_API_CHECK=1 LTX_BYPASS_MODEL_CHECK=1 vite", + "dev:debug": "cross-env BACKEND_DEBUG=1 ELECTRON_DEBUG=1 LTX_BYPASS_API_CHECK=1 LTX_BYPASS_MODEL_CHECK=1 vite", "typecheck": "concurrently \"pnpm run typecheck:ts\" \"pnpm run typecheck:py\"", "typecheck:ts": "tsc --noEmit", "typecheck:py": "cd backend && uv run pyright", From 510642c4795ec5168911fcfb779d296b3887b85c Mon Sep 17 00:00:00 2001 From: adunato Date: Wed, 1 Apr 2026 19:07:40 +0100 Subject: [PATCH 12/12] feat: bypass Electron first-run setup via LTX_BYPASS_MODEL_CHECK --- change-requests/CR002-developer-configuration/HLD.md | 9 +++++++-- electron/ipc/app-handlers.ts | 4 ++++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/change-requests/CR002-developer-configuration/HLD.md b/change-requests/CR002-developer-configuration/HLD.md index 1f9df583..385cac31 100644 --- a/change-requests/CR002-developer-configuration/HLD.md +++ b/change-requests/CR002-developer-configuration/HLD.md @@ -20,8 +20,13 @@ To bypass these gates, two environment variables were introduced: * **Location:** `backend/runtime_config/runtime_policy.py` (`decide_force_api_generations`) * **Logic:** If set to `"1"`, the function returns `False`, tricking the app into believing local generation is fully supported, thereby dismissing the mandatory API key prompt. * **`LTX_BYPASS_MODEL_CHECK`:** - * **Location:** `backend/ltx2_server.py` - * **Logic:** If set to `"1"`, the `REQUIRED_MODEL_TYPES` frozen set is overridden to be empty (`frozenset()`). This tricks the frontend's model status check into believing all necessary models are already downloaded, skipping the "Choose Location" installation screen. + * **Locations:** + * Backend: `backend/ltx2_server.py` + * Electron Main Process: `electron/ipc/app-handlers.ts` (`getSetupStatus`) + * **Logic:** + * In the backend, if set to `"1"`, the `REQUIRED_MODEL_TYPES` frozen set is overridden to be empty (`frozenset()`). This tricks the frontend's model status check (`/api/models/status`) into believing all necessary models are already downloaded. + * In the Electron main process, if set to `"1"`, the IPC handler immediately returns `{ needsSetup: false, needsLicense: false }`, overriding the local `app_state.json` file. + * **Impact:** Together, these bypass both the local app state checks and the backend filesystem checks, completely skipping the "Choose Location" installation screen. ### 2.2. Development Scripts configuration diff --git a/electron/ipc/app-handlers.ts b/electron/ipc/app-handlers.ts index f7eb3d27..e56ffe57 100644 --- a/electron/ipc/app-handlers.ts +++ b/electron/ipc/app-handlers.ts @@ -16,6 +16,10 @@ function getModelsPath(): string { } function getSetupStatus(settingsPath: string): { needsSetup: boolean; needsLicense: boolean } { + if (process.env.LTX_BYPASS_MODEL_CHECK === '1') { + return { needsSetup: false, needsLicense: false } + } + if (!fs.existsSync(settingsPath)) { return { needsSetup: true, needsLicense: true } }