-
Notifications
You must be signed in to change notification settings - Fork 3
Add a page on high level harnesses #9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
p3t3rzb
wants to merge
5
commits into
main
Choose a base branch
from
@p3t3rzb/high-level-harnesses
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+146
−0
Open
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
138 changes: 138 additions & 0 deletions
138
src/content/docs/expanding-horizons/high-level-harnesses.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,138 @@ | ||
| --- | ||
| title: High-level harnesses | ||
| description: Beyond individual agent sessions — scheduled automations, parallel agent fleets, and the emerging pattern of AI-driven code pipelines. | ||
| --- | ||
|
|
||
| import ExternalLink from "../../../components/ExternalLink.astro"; | ||
|
|
||
| The [harness engineering](/becoming-productive/harness-engineering/) chapter covered shaping a single agent's actions through AGENTS.md, skills, hooks, and subagents. | ||
| This page is one level of abstraction up — it covers tools and patterns that treat agents as a manageable workforce. | ||
|
|
||
| :::caution | ||
| Products and feature sets can change significantly between revisions of this guide. | ||
| Treat this page as an orientation, especially for building a solid intuition of the field, not a definitive reference. | ||
| ::: | ||
|
|
||
| ## From engineering to managing | ||
|
|
||
| So far in this guide, you have been an **engineer** — you worked interactively with a single agent, steering it turn by turn in real time. | ||
| Now, you will become a **manager**, delegating work to a fleet of agents running in parallel. | ||
| Instead of supervising each agent individually, you will manage the output queue — a review inbox, an issue tracker, a PR pipeline. | ||
| Your coding assistant no longer serves as a conductor, but as an orchestrator. | ||
|
|
||
| :::note[Remember] | ||
| The key shift is from "what should the agent do?" to "what work should be running right now, and how do I review what came back?" | ||
p3t3rzb marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ::: | ||
|
|
||
| ## Running agents in parallel | ||
|
|
||
| The key difference is running several agents simultaneously, each on an isolated task. | ||
p3t3rzb marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| You hand different issues to separate agents at once, come back and review, and merge the ones you like. | ||
| That is qualitatively different from the sequential, one-task-at-a-time conductor workflow from the previous chapters. | ||
|
|
||
| [Subagents](/becoming-productive/harness-engineering/#subagents) are also parallel, but they are different: a subagent is spawned **by the agent** to partition a single task's context. | ||
| The agent decides when to spawn one, waits for the result, and folds it back into its own session. | ||
| You as the human still trigger one top-level session and review one result. | ||
|
|
||
| What is described here is different: **you** spawn multiple fully independent agent sessions, each assigned to a separate task. | ||
| No session knows about the others. | ||
| You are not waiting on any one of them — you come back later and review the queue of results in bulk. | ||
|
|
||
| In practice, each agent needs its own isolated workspace — typically a separate Git worktree — so their changes do not interfere. | ||
| A dashboard or queue then surfaces results as agents finish, letting you review and merge at your own pace. | ||
|
|
||
| For example, <ExternalLink href="https://conductor.build/">Conductor</ExternalLink> is a tool built around this model, | ||
| running multiple AI coding agents (Claude Code and Codex) in parallel worktrees with a shared review dashboard. | ||
|
|
||
| ## Scheduled and recurring agents | ||
|
|
||
| Agents do not always need to wait for you to trigger them — you can set them up in advance to run on a schedule. | ||
| The pattern is similar to a cron job or a CI pipeline: describe a recurring task, define when it should run, and have an agent execute it in the background. | ||
| Results land in a review inbox or are auto-archived if nothing needs attention. | ||
|
|
||
| This is well-suited for tasks like: | ||
| - Daily issue triage | ||
| - Surfacing and summarizing CI failures | ||
| - Generating release briefs | ||
| - Checking for regressions between versions | ||
|
|
||
| With scheduled agents, the process becomes closer to a CI pipeline than a chat window — an agent is no longer a tool you reach for, but a background process. | ||
|
|
||
| As an example, OpenAI's <ExternalLink href="https://developers.openai.com/codex/app">Codex App</ExternalLink> includes an <ExternalLink href="https://developers.openai.com/codex/app/automations">Automations</ExternalLink> feature built around exactly this model. | ||
|
|
||
| ## Issue-tracker-driven orchestration | ||
|
|
||
| A natural extension of scheduled agents is wiring them directly to your issue tracker. | ||
| Instead of manually assigning tasks to agents, the system monitors a board and automatically spawns an agent for each new issue in scope. | ||
| Engineers decide what issues belong in scope; the orchestrator handles assignment and execution. | ||
|
|
||
| Agent behavior can be defined in a workflow file versioned alongside the code — the same way you version a CI pipeline. | ||
| When an agent finishes, it gathers evidence (CI results, PR review feedback, complexity analysis) for human review. | ||
|
|
||
| For example, <ExternalLink href="https://github.com/openai/symphony">Symphony</ExternalLink> is an open-source orchestration service published by OpenAI that implements this pattern, | ||
| monitoring a Linear board and running a Codex agent per issue in an isolated workspace. | ||
|
|
||
| :::tip | ||
| Issue-tracker-driven orchestration works best on codebases that have adopted [harness engineering](/becoming-productive/harness-engineering/). | ||
| ::: | ||
|
|
||
| ## Agent communication | ||
|
|
||
| Running multiple agents in parallel may cause coordination problem — agents must exchange information without overloading any one context window. | ||
| Two broad patterns have emerged. | ||
|
|
||
| The simpler one is **hub-and-spoke orchestration**, where a lead agent spawns workers, collects their outputs, and consolidates. | ||
| Workers never communicate directly. | ||
| The benefit is simplicity, as the full picture is present in one place. | ||
| The cost is that every intermediate result, log line, and failed attempt flows back through the orchestrator's context, degrading its reasoning quality over time. | ||
|
|
||
| The more capable pattern is **collaborative teaming**, where agents share a task list, claim work independently, and can send messages directly to one another. | ||
| A worker can flag a dependency, request a peer review, or broadcast a finding without routing it through the lead. | ||
| The lead's context stays clean; coordination happens at the edges. | ||
|
|
||
| In practice, most pipelines fall somewhere on a spectrum between these extremes, often organized into three levels: | ||
|
|
||
| 1. **Isolated workers** — each agent runs independently and returns its output to the caller. | ||
| 2. **Orchestrated workflows** — outputs become inputs for the next stage via shared files or aggregated results. | ||
| 3. **Collaborative teams** — agents share a task graph, can send direct or broadcast messages, and notify the lead when work completes. | ||
|
|
||
| The right level depends on how tightly coupled the tasks are. | ||
| Independent parallel tasks — security scan, test run, lint check — fit level 1 or 2 well. | ||
| Tasks that need to challenge or build on each other's intermediate findings call for level 3. | ||
|
|
||
| For reference, Claude Code Agent Teams implements level 3 with a shared task list, file-locked claiming, mailboxes for direct and broadcast messages, and idle notifications back to the lead. | ||
| Codex 0.117 introduced path-based agent addressing and structured inter-agent messaging for its multi-agent workflows. | ||
|
|
||
| ## The Code Factory pattern | ||
|
|
||
| Beyond specific products, there is an emerging pattern popularized by Ryan Carson under the name **Code Factory**. | ||
| The idea is a repository setup where agents autonomously write code, open pull requests, and a separate review agent validates those PRs with machine-verifiable evidence. | ||
| If validation passes, the PR merges without human intervention. | ||
|
|
||
| The continuous loop looks like this: | ||
|
|
||
| 1. Agent writes code and opens a PR. | ||
| 2. Risk-aware CI gates check the change. | ||
| 3. A review agent inspects the PR and collects evidence — screenshots, test results, static analysis. | ||
| 4. If all checks pass, the PR lands automatically. | ||
| 5. If anything fails, the agent retries or flags the issue for human review. | ||
|
|
||
| :::caution | ||
| A Code Factory is only as good as its quality gates. | ||
| An automated pipeline that merges bad PRs is strictly worse than one that does nothing. | ||
| Invest in solid tests, linters, and CI before automating the merge step. | ||
| ::: | ||
|
|
||
| - <ExternalLink href="https://x.com/ryancarson" /> | ||
|
|
||
| ## The one-human company | ||
|
|
||
| The Code Factory pattern is the technical foundation of a broader idea: that a single person with a well-configured agent fleet can operate at the scale that would previously have required a full engineering team. | ||
|
|
||
| This requires connecting agents to communication platforms, scheduling systems, and external services — turning a single machine into an always-on runtime that responds to messages, executes tasks, and ships work continuously. | ||
| As an example of tooling in this space, see <ExternalLink href="https://openclaw.ai/">OpenClaw</ExternalLink>, which packages infrastructure for exactly this kind of setup. | ||
|
|
||
| Steve Yegge, in a widely-read interview with The Pragmatic Engineer, argues that the engineering profession is reorganizing around exactly this spectrum. | ||
| His framing: most engineers are at the low end of AI adoption today, and those who stay there risk being outcompeted by engineers who learn to orchestrate agent fleets — to act as owners of work queues rather than writers of individual functions. | ||
|
|
||
| - <ExternalLink href="https://newsletter.pragmaticengineer.com/p/from-ides-to-ai-agents-with-steve" /> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.