diff --git a/docs/public/SUMMARY.md b/docs/public/SUMMARY.md index c9b8ba59..0502a1e7 100644 --- a/docs/public/SUMMARY.md +++ b/docs/public/SUMMARY.md @@ -19,6 +19,7 @@ * [Fleet Management](user-guide/fleet.md) * [Alerts](user-guide/alerts.md) * [Templates](user-guide/templates.md) +* [Shared Components](user-guide/shared-components.md) ## Operations diff --git a/docs/public/user-guide/pipeline-editor.md b/docs/public/user-guide/pipeline-editor.md index 00569111..1dfe6fc5 100644 --- a/docs/public/user-guide/pipeline-editor.md +++ b/docs/public/user-guide/pipeline-editor.md @@ -17,20 +17,19 @@ The editor is divided into four main areas: ## Component palette -The left sidebar lists every available Vector component, grouped into three sections: +The left sidebar has two tabs: -- **Sources** -- Components that ingest data into the pipeline. -- **Transforms** -- Components that process, filter, or reshape data in flight. -- **Sinks** -- Components that send data to downstream destinations. +- **Catalog** -- Lists every available Vector component, grouped by kind (Sources, Transforms, Sinks). Each section can be collapsed and is further organized by category (e.g. "Cloud Platform", "Aggregating", "Messaging"). +- **Shared** -- Lists [shared components](shared-components.md) available in the current environment. Filter by kind using the All / Source / Transform / Sink buttons. -Each section can be collapsed. When a section contains many components, they are further organized by category (e.g. "Cloud Platform", "Aggregating", "Messaging"). - -Use the **search bar** at the top of the palette to filter components by name, type, description, or category. +Use the **search bar** at the top of the palette to filter components by name, type, description, or category. The search applies to whichever tab is active. ### Adding a component Drag a component from the palette and drop it onto the canvas. A new node appears at the drop position, pre-configured with sensible defaults. You can also right-click the canvas to paste previously copied nodes. +Dragging a shared component from the **Shared** tab creates a **linked node** -- the node's configuration is pre-filled from the shared component and pinned to its current version. See [Shared Components](shared-components.md) for details on linked node behavior. + ## Canvas The canvas is where your pipeline takes visual shape. Each component is represented as a **node**, and data flow between components is represented as **edges** (connections). @@ -47,7 +46,7 @@ The canvas is where your pipeline takes visual shape. Each component is represen ### Context menus -- **Right-click a node** -- Opens a menu with Copy, Paste, Duplicate, and Delete actions. +- **Right-click a node** -- Opens a menu with Copy, Paste, Duplicate, Delete, and **Save as Shared Component** actions. The shared component option extracts the node's configuration into a reusable [shared component](shared-components.md) and links the node to it. - **Right-click an edge** -- Opens a menu with a Delete connection action. ## Node types @@ -116,6 +115,15 @@ The **Config** tab shows: - **Configuration form** -- Auto-generated form fields based on the component's configuration schema. Required fields are marked, and each field has contextual help. - **Secret picker** -- Sensitive fields (passwords, API keys, tokens) display a secret picker instead of a text input. You must select an existing secret or create a new one inline -- plaintext values cannot be entered directly into sensitive fields. See [Security](../operations/security.md#sensitive-fields) for details. +### Linked shared components + +When a node is linked to a [shared component](shared-components.md), the detail panel shows additional elements: + +- A **purple banner** with the shared component name and an "Open in Library" link. +- **Read-only configuration** -- All config fields are disabled. Edits must be made on the shared component's Library page. +- An **Unlink** button to convert the node back to a regular editable component (the current config snapshot is preserved). +- When the shared component has been updated since the node was last synced, an **amber update banner** appears with an "Accept update" button to pull in the latest configuration. + ### VRL editor for transforms For **remap**, **filter**, and **route** transforms, the detail panel includes an integrated VRL editor (powered by Monaco) instead of a plain text field. The VRL editor provides: diff --git a/docs/public/user-guide/pipelines.md b/docs/public/user-guide/pipelines.md index 30df9301..8dc23fc3 100644 --- a/docs/public/user-guide/pipelines.md +++ b/docs/public/user-guide/pipelines.md @@ -29,6 +29,7 @@ A pipeline moves through several states during its lifecycle: - **Stopped** -- The pipeline is deployed but all agent nodes have stopped processing it. - **Crashed** -- One or more agent nodes report that the pipeline has crashed. Check the pipeline logs for details. - **Pending deploy** -- Shown as an additional badge when the saved configuration differs from what is currently deployed. Deploy the pipeline to push the latest changes. +- **Updates available** -- Shown as an amber badge when the pipeline contains nodes linked to [shared components](shared-components.md) that have been updated. Hover over the badge to see which shared components have pending updates. Open the pipeline editor to review and accept the changes. - **Pending Approval** -- Shown when an editor has submitted a deploy request that is waiting for admin approval. See [Deploy approval workflows](#deploy-approval-workflows). ## Creating a pipeline diff --git a/docs/public/user-guide/shared-components.md b/docs/public/user-guide/shared-components.md new file mode 100644 index 00000000..fe30ccd6 --- /dev/null +++ b/docs/public/user-guide/shared-components.md @@ -0,0 +1,165 @@ +# Shared Components + +Shared components are reusable, environment-scoped pipeline component configurations that can be linked into multiple pipelines. Edit a shared component in one place and every linked pipeline sees the update -- no more hunting through dozens of pipelines to change an endpoint. + +## How shared components work + +A shared component stores a canonical configuration for a specific component type (e.g., an Elasticsearch sink with your production cluster settings). When you link a shared component into a pipeline, the pipeline node receives a snapshot of that configuration and pins to a specific version. + +When someone edits the shared component, its version number increments. Linked pipelines don't change automatically -- instead, they surface an **Update available** indicator at three levels: + +- **Pipeline list** -- An amber "Updates available" badge on the pipeline row. +- **Canvas** -- The linked node's footer changes to "Update available" with an amber dot. +- **Detail panel** -- A banner with an "Accept update" button. + +Pipeline owners review and explicitly accept updates before redeploying. This review-first model prevents surprise breakage across production pipelines. + +{% hint style="info" %} +Shared components are scoped to a single environment. A production Kafka config is separate from a staging one. If you need the same component in multiple environments, create a shared component in each. +{% endhint %} + +## Finding shared components + +Shared components are managed in the **Library** page, accessible from the sidebar. The Library has two sections: + +- **Templates** -- Reusable pipeline blueprints (see [Templates](templates.md)). +- **Shared Components** -- Reusable component configurations (this page). + +### Shared components list + +The list shows all shared components in the currently selected environment: + +| Column | Description | +|--------|-------------| +| **Name** | The shared component name. Click to open the detail page. | +| **Type** | The Vector component type (e.g., `elasticsearch`, `kafka`). | +| **Kind** | A badge indicating Source, Transform, or Sink. | +| **Linked Pipelines** | Number of distinct pipelines using this shared component. | +| **Version** | Current version number (increments on each config change). | +| **Last Updated** | When the configuration was last modified. | + +## Creating a shared component + +There are two ways to create a shared component. + +### From the Library page + +{% stepper %} +{% step %} +### Open the Library +Navigate to **Library > Shared Components** in the sidebar and click **New Shared Component**. +{% endstep %} +{% step %} +### Choose a component type +Select the component type from the catalog grid (e.g., Elasticsearch, Kafka, S3). +{% endstep %} +{% step %} +### Configure +Enter a **Name** and optional **Description**, then fill in the component configuration using the form editor. Click **Create** to save. +{% endstep %} +{% endstepper %} + +### From the pipeline editor + +{% stepper %} +{% step %} +### Right-click a node +In the pipeline editor, right-click any configured node on the canvas. +{% endstep %} +{% step %} +### Select Save as Shared Component +Choose **Save as Shared Component** from the context menu. A dialog opens. +{% endstep %} +{% step %} +### Name and save +Enter a **Name** and optional **Description**, then click **Save**. The node's current configuration becomes the shared component, and the node is automatically linked to it. +{% endstep %} +{% endstepper %} + +## Using shared components in pipelines + +### Adding from the component palette + +The component palette (left sidebar in the pipeline editor) has two tabs: + +- **Catalog** -- The standard list of all Vector component types. +- **Shared** -- Shared components available in the current environment. + +The Shared tab includes kind filter buttons (All, Source, Transform, Sink) and supports search by name or component type. Drag a shared component from the list onto the canvas to create a linked node with the shared configuration pre-filled. + +### Linked node appearance + +Linked nodes are visually distinct on the canvas: + +- **Purple accent border** -- A purple border and subtle glow distinguish linked nodes from regular ones. +- **Footer** -- Shows "Shared" with a link icon. +- **Stale indicator** -- When an update is available, the footer changes to "Update available" in amber, and a small amber dot appears on the node corner. + +### Detail panel for linked nodes + +When you select a linked node, the detail panel shows: + +- A **purple banner** with the shared component name and an "Open in Library" link. +- **Read-only configuration** -- Config fields are disabled because the configuration is managed centrally. To edit, go to the Library page. +- An **Unlink** button in the header to convert back to a regular editable node. + +{% hint style="warning" %} +Unlinking a node preserves its current configuration but breaks the connection to the shared component. Future updates to the shared component will not reach this node. +{% endhint %} + +## Editing a shared component + +{% stepper %} +{% step %} +### Open the detail page +Navigate to **Library > Shared Components** and click the shared component you want to edit. +{% endstep %} +{% step %} +### Modify the configuration +Update the configuration using the form editor. You can also edit the name and description. +{% endstep %} +{% step %} +### Save +Click **Save**. The version number increments automatically, and all linked pipelines will see an "Update available" indicator. +{% endstep %} +{% endstepper %} + +The detail page also shows a **Linked Pipelines** section listing every pipeline that uses this shared component, with a badge indicating whether each is up to date or has a pending update. + +## Accepting updates + +When a shared component is updated, linked pipeline nodes become stale. Pipeline owners decide when to accept the new configuration. + +### Single node + +Select the stale node in the pipeline editor. The detail panel shows an amber banner with an **Accept update** button. Click it to pull in the latest configuration and update the pinned version. + +### All nodes in a pipeline + +If a pipeline has multiple stale linked nodes, you can accept all updates at once from the pipeline's toolbar or through the API. + +{% hint style="info" %} +Accepting an update modifies the pipeline's saved configuration but does not automatically redeploy. You still need to deploy the pipeline to push the changes to agents. +{% endhint %} + +## Deleting a shared component + +On the shared component detail page, scroll to the **Danger Zone** section and click **Delete**. A confirmation dialog explains what will happen: + +- All linked pipeline nodes are **unlinked** -- their configurations remain intact, but they become regular standalone nodes. +- The shared component is permanently removed from the Library. + +{% hint style="danger" %} +Deleting a shared component cannot be undone. Linked nodes keep their last-synced configuration but lose the ability to receive future updates. +{% endhint %} + +## Edge cases + +| Scenario | Behavior | +|----------|----------| +| **Cloning a pipeline** | Linked nodes in the clone remain linked to the same shared components. | +| **Promoting a pipeline** | Shared component links are stripped during cross-environment promotion. Nodes retain their config snapshots as regular standalone nodes. | +| **Saving as template** | Templates are portable across environments, so shared component links are stripped. Template nodes store config snapshots only. | +| **Discarding changes** | Reverting to a previous version restores the shared component links as they were at that version. | +| **Copy/paste nodes** | Pasted nodes retain their link to the same shared component. | +| **GitOps / YAML export** | Generated YAML uses the resolved config snapshot. Importing from YAML creates regular unlinked nodes. | diff --git a/docs/public/user-guide/templates.md b/docs/public/user-guide/templates.md index 7aa23d46..49b72e4d 100644 --- a/docs/public/user-guide/templates.md +++ b/docs/public/user-guide/templates.md @@ -4,7 +4,7 @@ Templates are reusable pipeline blueprints that capture a complete pipeline grap ## Template library -The **Templates** page displays all templates available to your team as a card grid. Each card shows: +Templates are found in the **Library** page (accessible from the sidebar), under the **Templates** tab. The page displays all templates available to your team as a card grid. Each card shows: - **Name** -- The template name. - **Category** -- A label such as Logging, Metrics, Archival, Streaming, or a custom category you define. @@ -44,8 +44,8 @@ Click **Save Template**. The template now appears in the template library for al {% stepper %} {% step %} -### Open the Templates page -Navigate to **Templates** in the sidebar. Make sure you have an environment selected in the header. +### Open the Library +Navigate to **Library > Templates** in the sidebar. Make sure you have an environment selected in the header. {% endstep %} {% step %} ### Choose a template @@ -72,6 +72,7 @@ Templates are designed to be portable across environments, so they intentionally - **Secrets** -- Secret values (API keys, passwords, tokens) are never stored in templates. You must configure secrets in the target environment after creating a pipeline from a template. - **Certificates** -- TLS certificates are environment-specific and are not included. - **Environment bindings** -- Templates are not tied to any specific environment. They can be used in any environment within the team. +- **Shared component links** -- Nodes that are linked to [shared components](shared-components.md) have their links stripped when saved as a template. The node's configuration snapshot is preserved, but the template node becomes a standalone component. - **Deployment state** -- Pipelines created from templates start as drafts. You deploy them when ready. {% hint style="warning" %} diff --git a/prisma/schema.prisma b/prisma/schema.prisma index 01738443..19eb7b7f 100644 --- a/prisma/schema.prisma +++ b/prisma/schema.prisma @@ -133,6 +133,7 @@ model Environment { serviceAccounts ServiceAccount[] deployRequests DeployRequest[] teamDefaults Team[] @relation("teamDefault") + sharedComponents SharedComponent[] createdAt DateTime @default(now()) } @@ -370,6 +371,11 @@ model PipelineNode { positionX Float positionY Float disabled Boolean @default(false) + sharedComponentId String? + sharedComponent SharedComponent? @relation(fields: [sharedComponentId], references: [id], onDelete: SetNull) + sharedComponentVersion Int? + + @@index([sharedComponentId]) } enum ComponentKind { @@ -387,6 +393,25 @@ model PipelineEdge { sourcePort String? } +model SharedComponent { + id String @id @default(cuid()) + name String + description String? + componentType String + kind ComponentKind + config Json + version Int @default(1) + environmentId String + environment Environment @relation(fields: [environmentId], references: [id]) + createdAt DateTime @default(now()) + updatedAt DateTime @updatedAt + + linkedNodes PipelineNode[] + + @@unique([environmentId, name]) + @@index([environmentId]) +} + model PipelineVersion { id String @id @default(cuid()) pipelineId String diff --git a/src/app/(dashboard)/library/layout.tsx b/src/app/(dashboard)/library/layout.tsx new file mode 100644 index 00000000..575b6201 --- /dev/null +++ b/src/app/(dashboard)/library/layout.tsx @@ -0,0 +1,47 @@ +"use client"; + +import { usePathname, useRouter } from "next/navigation"; +import { cn } from "@/lib/utils"; +import { FileText, Link2 } from "lucide-react"; + +const libraryNavItems = [ + { title: "Templates", href: "/library/templates", icon: FileText }, + { title: "Shared Components", href: "/library/shared-components", icon: Link2 }, +]; + +export default function LibraryLayout({ children }: { children: React.ReactNode }) { + const pathname = usePathname(); + const router = useRouter(); + + return ( +
+ +
{children}
+
+ ); +} diff --git a/src/app/(dashboard)/library/page.tsx b/src/app/(dashboard)/library/page.tsx new file mode 100644 index 00000000..dae1da6c --- /dev/null +++ b/src/app/(dashboard)/library/page.tsx @@ -0,0 +1,5 @@ +import { redirect } from "next/navigation"; + +export default function LibraryPage() { + redirect("/library/templates"); +} diff --git a/src/app/(dashboard)/library/shared-components/[id]/page.tsx b/src/app/(dashboard)/library/shared-components/[id]/page.tsx new file mode 100644 index 00000000..30cf78e5 --- /dev/null +++ b/src/app/(dashboard)/library/shared-components/[id]/page.tsx @@ -0,0 +1,388 @@ +"use client"; + +import { useCallback, useState } from "react"; +import { useParams, useRouter } from "next/navigation"; +import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query"; +import { useTRPC } from "@/trpc/client"; +import { useEnvironmentStore } from "@/stores/environment-store"; +import { findComponentDef } from "@/lib/vector/catalog"; +import { toast } from "sonner"; +import Link from "next/link"; +import { + ArrowLeft, + ExternalLink, + Link2, + Loader2, + Save, + Trash2, +} from "lucide-react"; + +import { Button } from "@/components/ui/button"; +import { Badge } from "@/components/ui/badge"; +import { Input } from "@/components/ui/input"; +import { Label } from "@/components/ui/label"; +import { Textarea } from "@/components/ui/textarea"; +import { Skeleton } from "@/components/ui/skeleton"; +import { + Card, + CardContent, + CardDescription, + CardHeader, + CardTitle, +} from "@/components/ui/card"; +import { ConfirmDialog } from "@/components/confirm-dialog"; +import { SchemaForm } from "@/components/config-forms/schema-form"; + +/* ------------------------------------------------------------------ */ +/* Kind badge styling */ +/* ------------------------------------------------------------------ */ + +const kindVariant: Record = { + SOURCE: + "bg-emerald-100 text-emerald-800 dark:bg-emerald-900/40 dark:text-emerald-300", + TRANSFORM: + "bg-sky-100 text-sky-800 dark:bg-sky-900/40 dark:text-sky-300", + SINK: "bg-orange-100 text-orange-800 dark:bg-orange-900/40 dark:text-orange-300", +}; + +/* ------------------------------------------------------------------ */ +/* Page Component */ +/* ------------------------------------------------------------------ */ + +export default function SharedComponentDetailPage() { + const params = useParams<{ id: string }>(); + const router = useRouter(); + const trpc = useTRPC(); + const queryClient = useQueryClient(); + const selectedEnvironmentId = useEnvironmentStore( + (s) => s.selectedEnvironmentId, + ); + + const [name, setName] = useState(""); + const [description, setDescription] = useState(""); + const [config, setConfig] = useState>({}); + const [hasChanges, setHasChanges] = useState(false); + const [initialized, setInitialized] = useState(false); + const [deleteOpen, setDeleteOpen] = useState(false); + + const componentQuery = useQuery( + trpc.sharedComponent.getById.queryOptions( + { id: params.id, environmentId: selectedEnvironmentId! }, + { + enabled: !!selectedEnvironmentId && !!params.id, + }, + ), + ); + + const sc = componentQuery.data; + + // Initialize form state from fetched data + if (sc && !initialized) { + setName(sc.name); + setDescription(sc.description ?? ""); + setConfig((sc.config as Record) ?? {}); + setInitialized(true); + } + + const componentDef = sc + ? findComponentDef(sc.componentType, sc.kind.toLowerCase() as "source" | "transform" | "sink") + : undefined; + + const handleNameChange = useCallback((val: string) => { + setName(val); + setHasChanges(true); + }, []); + + const handleDescriptionChange = useCallback((val: string) => { + setDescription(val); + setHasChanges(true); + }, []); + + const handleConfigChange = useCallback((vals: Record) => { + setConfig(vals); + setHasChanges(true); + }, []); + + const updateMutation = useMutation( + trpc.sharedComponent.update.mutationOptions({ + onSuccess: () => { + toast.success("Shared component updated"); + setHasChanges(false); + queryClient.invalidateQueries({ + queryKey: trpc.sharedComponent.getById.queryKey(), + }); + queryClient.invalidateQueries({ + queryKey: trpc.sharedComponent.list.queryKey(), + }); + }, + onError: (err) => { + toast.error(err.message); + }, + }), + ); + + const deleteMutation = useMutation( + trpc.sharedComponent.delete.mutationOptions({ + onSuccess: () => { + toast.success("Shared component deleted"); + router.push("/library/shared-components"); + }, + onError: (err) => { + toast.error(err.message); + }, + }), + ); + + const handleSave = () => { + if (!sc || !selectedEnvironmentId) return; + updateMutation.mutate({ + id: sc.id, + environmentId: selectedEnvironmentId, + name, + description: description || null, + config, + }); + }; + + const handleDelete = () => { + if (!sc || !selectedEnvironmentId) return; + deleteMutation.mutate({ id: sc.id, environmentId: selectedEnvironmentId }); + }; + + if (!selectedEnvironmentId) { + return ( +
+
+ Select an environment from the header to view this component +
+
+ ); + } + + if (componentQuery.isLoading) { + return ( +
+ +
+
+ + +
+
+ + +
+
+
+ ); + } + + if (!sc) { + return ( +
+
+ Shared component not found +
+
+ ); + } + + return ( +
+ {/* Back + Header */} +
+ + + Back to Shared Components + + +
+
+
+

{sc.name}

+ + {sc.kind} + + v{sc.version} +
+

+ {sc.componentType} · {sc.linkedPipelines.length} linked{" "} + {sc.linkedPipelines.length === 1 ? "pipeline" : "pipelines"} +

+
+ +
+
+ + {/* Content */} +
+ {/* Left column */} +
+ {/* Details card */} + + + Details + + +
+ + handleNameChange(e.target.value)} + /> +
+
+ +