Skip to content

Surface LLM errors in CLI, use pi-ai credential detection, and implement model fallback#2

Open
Priyanshu-Priyam wants to merge 2 commits intoopen-gitagent:mainfrom
Priyanshu-Priyam:fix/surface-llm-errors-and-model-fallback
Open

Surface LLM errors in CLI, use pi-ai credential detection, and implement model fallback#2
Priyanshu-Priyam wants to merge 2 commits intoopen-gitagent:mainfrom
Priyanshu-Priyam:fix/surface-llm-errors-and-model-fallback

Conversation

@Priyanshu-Priyam
Copy link

Summary

  • Surface LLM errors in CLI: When a model returns stopReason: "error" (e.g. Bedrock rejecting a model ID), the CLI now prints the error message in red instead of showing a blank line. Also fires on_error hooks and audit logging for these failures.

  • Use pi-ai's getEnvApiKey() for credential detection: Replaces the hardcoded 6-provider API key map with pi-ai's provider-aware function, which supports all providers including Amazon Bedrock, Google Vertex, Azure OpenAI, GitHub Copilot, and others.

  • Implement model.fallback support: When the preferred model fails, automatically retry with the next model from agent.yaml's model.fallback list. Works in both CLI (REPL + single-shot) and SDK (query()) paths. This activates the already-declared but previously unused fallback field in the agent manifest.

Motivation

When using amazon-bedrock:anthropic.claude-opus-4-6-v1, Bedrock returns an error ("on-demand throughput isn't supported, use an inference profile"). The CLI showed a blank response with no error -- making it impossible to diagnose. The SDK path (sdk.ts) already handled this correctly; the CLI did not.

Test plan

  • npm run build passes
  • All 19 existing tests pass
  • Manual: run with a model that returns an error -- error now visible in red
  • Manual: set fallback in agent.yaml with a broken preferred model -- agent falls back automatically

Priyanshu-Priyam... added 2 commits March 6, 2026 21:33
…ent model fallback

Three related fixes for silent model failures:

1. CLI now prints LLM-level errors (stopReason: "error") instead of showing
   a blank line. Fires on_error hooks and audit logging for these failures.

2. Replace hardcoded 6-provider API key map with pi-ai's getEnvApiKey(),
   which supports all providers including Bedrock, Vertex, Azure, etc.

3. Implement model fallback: when the preferred model fails, automatically
   retry with models from agent.yaml's model.fallback list. Works in both
   CLI (REPL + single-shot) and SDK (query()) paths.

Made-with: Cursor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant