Improve agent loop: multi-turn context and Claude Agent SDK

[baryhuang]

Current agent loop

Simple single-turn ReAct loop (agent-runtime.ts):

1. Capture screenshot → scale → send to model as fresh single-turn API call
2. Model returns one action → execute via adb → append result to history
3. Repeat until finish or max steps

What the model sees each step: system prompt + one screenshot + last 8 history strings (flat text, one line per step).

Limitations

• No multi-turn conversation — each step is an independent API call, no memory of prior model responses
• No state accumulation — model must re-derive its plan every step from a screenshot and 8 terse history lines
• No tool call chaining — exactly one action per API round-trip
• Fixed context — same system prompt every step, no dynamic progress tracking

This works for simple tasks but causes inefficiency on complex multi-step tasks (e.g. GPT-5.2 needs ~30% more steps than Claude Sonnet 4.5 on the same task).

Proposal: Claude Agent SDK

Use Claude Agent SDK (Claude Code) as the agent driver. This would give us:

• Multi-turn conversation — model retains full context across steps
• Native tool use — agent SDK manages the tool call loop, retries, and error handling
• Built-in agentic patterns — planning, reflection, and recovery without manual prompt engineering
• Streaming — real-time progress instead of waiting for full responses

The current tool definitions (src/agent/tools.ts) are already compatible with the SDK's tool format.

[tsubasakong]

/fix

[github-actions]

Below is a concrete fix recommendation based on the current code (checked src/agent/agent-runtime.ts, src/agent/model-client.ts, src/agent/prompts.ts, src/agent/tools.ts).

Probable root causes

• Agent loop is intentionally stateless: every step rebuilds a fresh system+user prompt and calls the model once (AgentRuntime.runTask → ModelClient.nextStep). There is no persisted model-side conversation state, only a short text history (buildUserPrompt keeps the last 8 history lines).
• Tooling is forced to exactly one tool per API call (tool_choice: "required"), so there’s no multi-tool chaining or model-driven internal loop.
• Context is flattened into a single user prompt string; no assistant/tool messages are preserved as structured chat turns, so the model can’t carry forward any latent plan beyond those 8 lines.

Likely files/modules involved

• src/agent/agent-runtime.ts (main loop; history window; snapshot handling; step logging)
• src/agent/model-client.ts (single-call API usage; tool parsing; no conversation state)
• src/agent/prompts.ts (history truncation and user prompt construction)
• src/agent/tools.ts (tool schema; current “one tool per step” assumption)
• src/config/index.ts (agent config flags if adding new driver/feature flag)

Reproduction / verification steps

1. Run a multi-step task that requires intermediate sub-goals (e.g., “Open Gmail, draft an email, attach the latest screenshot, and save as draft”).
2. Observe step count and repeated re-derivation in session logs (sessions/.../steps.jsonl).
3. Verify the prompt only includes last 8 history lines (see buildUserPrompt), and each step issues a new single API call (see ModelClient.nextStep).
4. Compare step count vs a multi-turn agent loop (Claude Agent SDK or any loop that maintains chat state).

Minimal fix options (recommended approach first)

1. Recommended: Add a multi-turn chat state + tool loop inside ModelClient (keep current architecture)

• Maintain an in-memory conversation for the session (system + user + assistant tool calls + tool results). Push each step’s tool call and execution result as structured messages.
• Switch tool_choice to "auto" and implement a tool-call loop that consumes all tool calls returned in a single response (for both chat and responses mode). This enables chainable actions in one model turn.
• Preserve the existing history string for logging/debug only, but use the chat history as the primary context.
• Add a config flag: agent.driver = react|chat_loop (default to current behavior for safe rollback).

2. Alternative: Integrate Claude Agent SDK as a second driver

• Create a new driver module (e.g., src/agent/claude-agent.ts) and wire it in AgentRuntime.runTask behind a config flag.
• Map existing tools (CHAT_TOOLS/RESPONSES_TOOLS) to Claude’s tool spec and let SDK handle the tool loop.
• Requires Anthropic API wiring and an additional dependency; adds operational complexity and two agent paths to maintain.

3. Partial incremental: Increase history window and include full structured step records

• Expand history beyond 8 lines and include serialized tool args and results. This helps but does not solve single-turn tool chaining and still lacks conversational memory.

Test cases to add/update

• Unit test: ModelClient tool-loop processing handles multiple tool calls and preserves message order.
• Integration test: a multi-step task that requires remembering earlier decisions (e.g., choose a settings option, then return and confirm it).
• Regression test: ensure current single-tool behavior still works when agent.driver=react.
• Safety test: tool-call loop terminates (max tool calls per turn, max retries).

Risk and rollback notes

• Risks: increased token usage/cost; longer latency; potential for tool-loop runaway if not capped; memory/privacy exposure if full history is stored.
• Mitigations: add max tool calls per turn, max conversation length with truncation, and a feature flag for easy rollback.
• Rollback: switch config back to react (current single-step loop) without code removal.

Lightweight checks I ran

• Reviewed the current loop and prompt construction to confirm history truncation and single-call behavior in AgentRuntime.runTask and ModelClient.nextStep.

Focused follow-up questions

• Do you want the Claude Agent SDK to fully replace the OpenAI driver, or live alongside it behind a config flag?
• Any constraints on adding a new dependency and Anthropic API keys in config?
• Desired acceptance criteria for “multi-turn memory” (e.g., minimum steps saved or specific task benchmarks)?

If you want, I can sketch the driver interface and a minimal tool-loop implementation next.

Generated by issue-fix for issue #4

[m13v]

multi-turn context with tool use is one of those things that seems straightforward but has a lot of edge cases. the main thing we learned is to keep the tool results compact - if a tool returns a huge response, summarize it before appending to context. also worth implementing a token budget tracker so the agent knows when it is running low and can proactively summarize older turns instead of hitting the limit mid-conversation.

[m13v]

here's the chat provider where we handle multi-turn context with token tracking: https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatProvider.swift - and the tool executor for compact results: https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatToolExecutor.swift