Separate Responsibility & Multi-Agent Pattern for Spec Generation

Multi-Agent Spec Generation

Refactor Phase 2 (spec generation) from a single spec_agent into specialized agents
running in parallel, each responsible for one evaluator type. This reduces per-agent
context, improves focus, and enables independent retry on failure.

Proposed agents

Agent	Output	Responsibility
dataset_agent	list[DatasetCase]	Converts verified exploration results into inputs + expected + raises cases. Straightforward mapping — minimal LLM reasoning.
assertion_agent	list[AssertionCase]	Writes global property-based assertions from function semantics: output >= 0, len(output) == len(input), all(isinstance(x, int) for x in output), etc. Only assertions, no dataset.
type_agent	list[IsInstanceCase]	Determines return type checks, strict_type decisions, per-case type overrides. Uses real type observations from exploration.
composer_agent	EvalsSource (final YAML)	Receives outputs from the three agents above → merges into valid YAML spec. Only job is structure + validation, no eval logic.

Execution flow

Phase 1: Exploration (unchanged)
  └─ explorer_agent → snippets → executor → verified results

Phase 2: Multi-Agent Spec Generation
  ┌─ dataset_agent    ──┐
  ├─ assertion_agent  ──┤  (parallel)
  ├─ type_agent       ──┤
  └────────────────────→ composer_agent → YAML spec

Benefits over single-agent

• Less context per agent: each agent gets only the eval spec rules for its evaluator type
• Parallel execution: dataset, assertion, and type agents run concurrently
• Targeted retry: if assertion agent fails, only it retries — dataset stays
• Better quality: focused agents produce better assertions than a jack-of-all-trades
• Composable: easy to add new agents (e.g. duration_agent, pattern_agent) without touching the others

Design considerations

• Each agent output should be a Pydantic model for type safety
• composer_agent does validate_and_fix_spec() as final step
• Exploration results are shared context for all agents (read-only)
• Agent-level logfire spans for observability
• Fallback: if any agent errors, composer_agent should still produce a valid (partial) spec