Operational pipelines for Claude Code
Readable agents. Inspectable state. Predictable costs.
Building a pipeline with Claude Code? Start with CLAUDE.md.
Claude Code is great for tasks. But when you need multi-step pipelines — research → analyze → generate, batch processing overnight, workflows that run unattended — you hit gaps:
- No state persistence between steps
- No resume after failure
- No coordination for parallel workers
- No cost control
claude-pipe is a set of conventions that fills these gaps. File-based state you can cat. Flat orchestration you can debug. Agents you can git blame.
Claude Code has architectural constraints. claude-pipe works with them:
| Constraint | Why It Exists | Convention |
|---|---|---|
| Subagents can't spawn subagents | Forces flat hierarchy → easier debugging | Manager = Skill (ROOT reads instructions) |
| Context is finite | Prevents runaway costs | Workers write to files, not context |
| Model is "blind" until Read() | Explicit I/O = traceable | File-based state: cat state.yaml |
| Skills are steering, not code | Flexible, not brittle | Conventions over enforcement |
Result: Flat orchestration, file-based state, predictable costs.
| Problem | Without claude-pipe | With claude-pipe |
|---|---|---|
| Pipeline fails at step 4 | Re-run from scratch | Resume from step 4 |
| Agent loops forever | $50 bill | Circuit breaker stops it |
| "What happened?" | Dig through logs | cat artifacts/state.yaml |
| Share with team | Copy-paste prompts | git push |
ROOT orchestrates, Workers execute, Files persist.
┌─────────────────────────────────────────────────────┐
│ USER │
│ (slash commands) │
└─────────────────────┬───────────────────────────────┘
│
┌─────────────────────▼───────────────────────────────┐
│ ROOT │
│ (Claude Code CLI) │
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Skill │ │ Skill │ │ Skill │ Skills │
│ │(manager)│ │(atomic) │ │(domain) │ (libs) │
│ └─────────┘ └─────────┘ └─────────┘ │
│ │
│ ┌─────────────────────────────────────────────┐ │
│ │ Task tool │ │
│ │ (spawn subagents) │ │
│ └─────────────────────────────────────────────┘ │
└─────────────────────┬───────────────────────────────┘
│
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ Worker │ │ Worker │ │ Worker │
│ (subagent)│ │ (subagent)│ │ (subagent)│
└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
│ │ │
▼ ▼ ▼
┌───────────────────────────────────────────┐
│ FILES │
│ (L0 → L1 → L2 → L3) │
│ (persistent state layer) │
└───────────────────────────────────────────┘
Key insight: Manager is a Skill loaded into ROOT (not a separate agent). ROOT reads manager instructions and executes them directly. This is how you get orchestration despite the flat hierarchy constraint.
State flows through four layers:
L0 (Config) → L1 (Directives) → L2 (Operational) → L3 (Artifacts)
user_profile.yaml plan.yaml aspects/*.yaml FINAL_REPORT.md
directives.yaml synthesis.yaml
| Layer | What | Mutability | Example |
|---|---|---|---|
| L0 | User config | User-edited | user_profile.yaml |
| L1 | Session plan | Agent-generated, user-approved | plan.yaml |
| L2 | Working data | Agent-generated | aspects/*.yaml |
| L3 | Final outputs | Append-only | FINAL_REPORT.md |
Contract: L1 constraints always override L2 decisions. L3 must trace to L2 sources.
Semantic decomposition → parallel research → synthesis.
┌─────────────┐ ┌─────────────────┐ ┌───────────┐ ┌───────────┐
│ Planning │────▶│ Parallel Research│────▶│ Synthesis │────▶│ Report │
│ (1 agent) │ │ (N agents) │ │ (1 agent) │ │ │
└─────────────┘ └─────────────────┘ └───────────┘ └───────────┘
│ │ │ │
▼ ▼ ▼ ▼
plan.yaml aspects/*.yaml synthesis.yaml FINAL_REPORT.md
State file enables resume:
# artifacts/{session}/state.yaml
current_phase: synthesis
phase_states:
planning: completed
research: completed
synthesis: in_progress
workers:
aspect_1: completed
aspect_2: completed
aspect_3: failed # ← Resume re-runs only thisData partitioning → parallel classification → aggregation.
┌─────────────┐ ┌─────────────────┐ ┌───────────┐
│ Partition │────▶│ Classify ×N │────▶│ Aggregate │
│ │ │ (parallel) │ │ │
└─────────────┘ └─────────────────┘ └───────────┘
│ │ │
▼ ▼ ▼
manifest.yaml batches/*.yaml taxonomy.yaml
Use when: Large datasets that can be processed independently per batch.
Codebase scanning → parallel module documentation → assembly → docs site.
┌─────────┐ ┌─────────────────┐ ┌──────────┐ ┌─────────┐ ┌──────────┐
│ Scan │────▶│ Document ×N │────▶│ Assemble │────▶│ Quality │────▶│ Emit ×M │
│ │ │ (per module) │ │ │ │ Gate │ │(per file) │
└─────────┘ └─────────────────┘ └──────────┘ └─────────┘ └──────────┘
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
plan.yaml modules/*.yaml assembly.yaml quality.yaml docs/*.md
Pattern: Subagent = Module (structural partitioning). Double fan-out: once per module, once per output file. Zero external deps.
Reconnaissance → parallel vulnerability scanning → triage → severity-ranked report.
┌─────────┐ ┌─────────────────┐ ┌──────────┐ ┌─────────┐ ┌──────────┐
│ Recon │────▶│ Scan ×7 │────▶│ Triage │────▶│ Quality │────▶│ Report │
│ │ │(per vuln category)│ │ │ │ Gate │ │ │
└─────────┘ └─────────────────┘ └──────────┘ └─────────┘ └──────────┘
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
recon.yaml findings/*.yaml triage.yaml quality.yaml SECURITY_REPORT.md
Pattern: Subagent = Category (domain expertise partitioning). Each scanner is an expert in one OWASP category. Zero external deps.
When N tasks are independent:
## Manager Skill Instructions
For each aspect in plan.aspects:
Task(
subagent_type: "general-purpose",
prompt: "Load aspect-researcher. Research: {aspect.name}",
run_in_background: true
)
Wait for all tasks to complete.Key: All Task calls in ONE message = parallel execution.
When each phase depends on the previous:
Phase 1 → Gate → Phase 2 → Gate → Phase 3
│ │ │
▼ ▼ ▼
plan.yaml aspects/*.yaml synthesis.yaml
Gates = transition conditions:
gate:
type: file_exists
condition: "plan.yaml exists"
gate:
type: quality_threshold
condition: "min 3 aspects completed"Skills are knowledge libraries, not agents. They inject instructions into context.
Agent A + grounding-protocol = Agent A that doesn't hallucinate
Agent B + grounding-protocol = Agent B that doesn't hallucinate
| Type | Purpose | Example |
|---|---|---|
| atomic | Single operation | tier-weights, slop-check |
| composite | Combined operations | source-evaluation |
| domain | Knowledge pack | training-science |
| manager | Workflow instructions | manager-research |
Manager skill = orchestration instructions that ROOT executes directly.
.claude/
├── agents/ # Agent definitions (markdown)
└── skills/ # Skill definitions (markdown)
artifacts/
└── {session}/ # Runtime outputs (gitignored)
├── state.yaml # Pipeline state
└── ... # Phase outputs
For Claude Code: Read CLAUDE.md — it has the step-by-step recipe for building a pipeline.
For humans: Read the Quickstart Guide — annotated walkthrough with decision trees and a minimal example.
To run an example:
cd examples/research-pipeline
/manager-research "your topic"
Good fit:
- Multi-step research pipelines
- Data processing: gather → analyze → generate
- Workflows with phases and quality gates
- Overnight / unattended runs
- Teams sharing agent conventions via git
Not a good fit:
- Simple single-agent tasks
- Real-time chat applications
- Tasks without intermediate state
- Sub-millisecond latency requirements
- ROOT orchestrates, Workers execute — no nested spawns
- Skills are shared libraries — knowledge reuse
- Files are state — everything persists to YAML
- Layers have contracts — L1 constrains L2, L2 feeds L3
- Manager = Skill — workflow instructions for ROOT
| Primitive | What It Does |
|---|---|
Task(run_in_background: true) |
Spawn parallel worker |
TaskOutput(block: true) |
Wait for worker |
Skill("manager-x") |
Load orchestration instructions |
state.yaml |
Pipeline state for resume |
| Convention | Why |
|---|---|
| Workers write to files | Context stays small, state persists |
| Manager is a Skill | ROOT must control spawning |
| Gates between phases | Quality control, cost control |
| Session directories | Isolate runs, enable replay |
Running pipelines on OpenClaw (or other alternative Claude frontends)? See OPENCLAW.md for the adaptation guide — only fan-out phases need changes.
See CONTRIBUTING.md.
MIT — see LICENSE.