A multi-provider LLM framework for Rust. Build type-safe chat clients with tool calling, structured output, streaming, and embeddings — swap providers with a single line change.
- Multi-provider — Gemini, Claude, OpenAI, and Router today, more coming (see Roadmap)
- Router — route requests across multiple providers with fallback and custom strategies (keyword, embedding, capability-based)
- Type-safe builder — compile-time enforcement of valid configurations via type-state pattern
- Tool calling — define tools with
#[tool], the framework handles the call loop automatically - Structured output — deserialize model responses directly into your Rust types via
schemars - Streaming — real-time token-by-token output with tool call support
- Human in the loop — pause mid-turn on sensitive tool calls, let a human approve or reject, then resume the stream
- Embeddings — generate vector embeddings through the same unified API
- Retry & callbacks — configurable retry strategies with before/after hooks
- Native tools — provider-specific features like Google Search, code execution, web search
Add to your Cargo.toml:
[dependencies]
chat-rs = { version = "0.1.1", features = ["openai"] }
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }use chat_rs::{ChatBuilder, openai::OpenAIBuilder, types::messages};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let client = OpenAIBuilder::new().with_model("gpt-4o-mini").build();
let mut chat = ChatBuilder::new().with_model(client).build();
let mut messages = messages::from_user(vec!["Hey there!"]);
let res = chat.complete(&mut messages).await?;
println!("{:?}", res.content);
Ok(())
}Set your API key via environment variable (OPENAI_API_KEY, GEMINI_API_KEY, or CLAUDE_API_KEY), or pass it explicitly with .with_api_key().
Enable providers via feature flags:
# Pick one or more
chat-rs = { version = "0.1.1", features = ["gemini"] }
chat-rs = { version = "0.1.1", features = ["claude"] }
chat-rs = { version = "0.1.1", features = ["openai"] }
chat-rs = { version = "0.1.1", features = ["router", "gemini", "claude"] }
chat-rs = { version = "0.1.1", features = ["gemini", "claude", "openai", "stream"] }| Provider | Feature | API Key Env Var | Builder |
|---|---|---|---|
| Google Gemini | gemini |
GEMINI_API_KEY |
GeminiBuilder |
| Anthropic Claude | claude |
CLAUDE_API_KEY |
ClaudeBuilder |
| OpenAI | openai |
OPENAI_API_KEY |
OpenAIBuilder |
| Router | router |
— | RouterBuilder |
Swapping providers is a one-line change — replace the builder, everything else stays the same:
// Gemini
let client = GeminiBuilder::new()
.with_model("gemini-2.5-flash".to_string())
.build();
// Claude
let client = ClaudeBuilder::new()
.with_model("claude-sonnet-4-20250514".to_string())
.build();
// OpenAI
let client = OpenAIBuilder::new()
.with_model("gpt-4o")
.build();
// Same from here on
let mut chat = ChatBuilder::new().with_model(client).build();Define tools with the #[tool] macro from tools-rs and register them with collect_tools(). The framework automatically loops through tool calls until the model is done.
use chat_rs::{ChatBuilder, gemini::GeminiBuilder, types::messages::content};
use tools_rs::{collect_tools, tool};
#[tool]
/// Looks up the current weather for a given city.
async fn get_weather(city: String) -> String {
format!("The weather in {} is sunny, 22°C", city)
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let client = GeminiBuilder::new()
.with_model("gemini-2.5-flash".to_string())
.build();
let tools = collect_tools();
let mut chat = ChatBuilder::new()
.with_tools(tools)
.with_model(client)
.with_max_steps(5)
.build();
let mut messages = messages::Messages::default();
messages.push(content::from_user(vec!["What's the weather in Tokyo?"]));
let response = chat.complete(&mut messages).await.map_err(|e| e.err)?;
println!("{:?}", response.content);
Ok(())
}Deserialize model responses directly into typed Rust structs. Your type must derive JsonSchema and Deserialize.
use schemars::JsonSchema;
use serde::Deserialize;
#[derive(JsonSchema, Deserialize, Clone, Debug)]
struct User {
pub name: String,
pub likes: Vec<String>,
}
let mut chat = ChatBuilder::new()
.with_structured_output::<User>()
.with_model(client)
.build();
let response = chat.complete(&mut messages).await?;
println!("Name: {}, Likes: {:?}", response.content.name, response.content.likes);Enable the stream feature flag:
chat-rs = { version = "0.1.1", features = ["gemini", "stream"] }use chat_rs::StreamEvent;
use futures::StreamExt;
let mut chat = ChatBuilder::new()
.with_model(client)
.build();
let mut stream = chat.stream(&mut messages).await?;
while let Some(chunk) = stream.next().await {
match chunk? {
StreamEvent::TextChunk(text) => print!("{}", text),
StreamEvent::ReasoningChunk(thought) => print!("[thinking] {}", thought),
StreamEvent::ToolCall(fc) => println!("[calling {}]", fc.name),
StreamEvent::ToolResult(fr) => println!("[tool returned]"),
StreamEvent::Done(_) => break,
}
}Mark tools that need human approval via #[tool] metadata and supply a strategy closure. When the model calls such a tool, chat.stream() yields StreamEvent::Paused(PauseReason) and terminates. Resolve the pending tools on messages (approve or reject), then call stream() again — the core loop picks up where it left off.
use chat_rs::{Action, ChatBuilder, ScopedCollection, StreamEvent, PauseReason};
use tools_rs::{FunctionCall, ToolCollection, tool};
use serde::Deserialize;
#[derive(Debug, Default, Clone, Deserialize)]
#[serde(default)]
struct ApprovalMeta { requires_approval: bool }
#[tool(requires_approval = true)]
/// Sends an email.
async fn send_email(to: String, subject: String) -> String {
format!("sent to {to}: {subject}")
}
fn strategy(_call: &FunctionCall, meta: &ApprovalMeta) -> Action {
if meta.requires_approval { Action::RequireApproval } else { Action::Execute }
}
let tools: ToolCollection<ApprovalMeta> = ToolCollection::collect_tools()?;
let scoped = ScopedCollection::new(tools, strategy);
let mut chat = ChatBuilder::new()
.with_model(client)
.with_scoped_tools(scoped)
.build();
let mut stream = chat.stream(&mut messages).await?;
while let Some(evt) = stream.next().await {
match evt? {
StreamEvent::TextChunk(t) => print!("{t}"),
StreamEvent::Paused(PauseReason::AwaitingApproval { tool_ids }) => {
for id in tool_ids {
if let Some(tool) = messages.find_tool_mut(&id) {
tool.approve(None); // or tool.reject(Some("denied".into()))
}
}
break;
}
_ => {}
}
}
// Call chat.stream(&mut messages) again to resume the same turn.See examples/claude/hitl.rs, examples/openai/hitl.rs, and examples/gemini/hitl.rs for full interactive REPLs.
let client = GeminiBuilder::new()
.with_model("gemini-embedding-001".to_string())
.with_embeddings(Some(768))
.build();
let mut chat = ChatBuilder::new()
.with_model(client)
.with_embeddings()
.build();
let response = chat.embed(&mut messages).await?;
println!("{:?}", response.embeddings);Provider-specific capabilities beyond standard tool calling:
// Gemini: Google Search, Code Execution, Google Maps
let client = GeminiBuilder::new()
.with_model("gemini-2.5-flash".to_string())
.with_google_search()
.with_code_execution()
.build();
// OpenAI: Web Search
let client = OpenAIBuilder::new()
.with_model("gpt-4o")
.with_web_search(Some(SearchContextSizeEnum::High), None)
.build();Use local or proxy servers that implement the OpenAI Responses API:
let client = OpenAIBuilder::new()
.with_model("llama3")
.with_custom_url("http://localhost:11434/v1".to_string())
.with_api_key("ollama".to_string())
.build();Note: The custom endpoint must support the Responses API format (
POST /responses), not the Chat Completions API.
Route requests across multiple providers with automatic fallback on retryable errors. Add a custom RoutingStrategy to control provider selection based on keywords, embeddings, capabilities, or any logic you need.
use chat_rs::{
ChatBuilder,
router::RouterBuilder,
gemini::GeminiBuilder,
claude::ClaudeBuilder,
types::messages,
};
let gemini = GeminiBuilder::new()
.with_model("gemini-2.5-flash".to_string())
.build();
let claude = ClaudeBuilder::new()
.with_model("claude-sonnet-4-20250514".to_string())
.build();
let router = RouterBuilder::new()
.add_provider(gemini)
.add_provider(claude)
// .with_strategy(my_strategy) // optional custom routing
// .circuit_breaker(CircuitBreakerConfig::default()) // optional circuit breaker
.build();
let mut chat = ChatBuilder::new().with_model(router).build();
let mut msgs = messages::from_user(vec!["Hello!"]);
let res = chat.complete(&mut msgs).await?;Without a custom strategy, the router tries providers in order and falls back on retryable errors (rate limits, network issues). Non-retryable errors are returned immediately.
Enable the optional circuit breaker to automatically skip providers that have failed repeatedly, and probe them again after a configurable recovery timeout:
use chat_rs::router::CircuitBreakerConfig;
let router = RouterBuilder::new()
.add_provider(gemini)
.add_provider(claude)
.circuit_breaker(CircuitBreakerConfig {
failure_threshold: 3,
recovery_timeout: std::time::Duration::from_secs(30),
})
.build();Streaming is also supported via StreamRouterBuilder — enable the stream feature flag and use providers that implement ChatProvider.
Providers are generic over a pluggable Transport trait. The default transport is ReqwestTransport (HTTP via reqwest) — it's used automatically when you call .build() on any builder.
To share an HTTP client across providers:
use chat_rs::openai::{OpenAIBuilder, ReqwestTransport};
let http = ReqwestTransport::from(my_reqwest_client);
let client = OpenAIBuilder::new()
.with_model("gpt-4o")
.with_transport(http.clone()) // Clone shares the connection pool
.build();To use WebSocket transport (e.g. for OpenAI's Responses API over WS):
chat-rs = { version = "0.1.1", features = ["openai", "stream", "tokio-tungstenite"] }use chat_rs::{openai::OpenAIBuilder, transport::AsyncWsTransport};
let ws = AsyncWsTransport::new()
.with_message_type("response.create"); // OpenAI WS envelope
let client = OpenAIBuilder::new()
.with_model("gpt-4o")
.with_transport(ws)
.build();Two WebSocket transports are available, feature-gated:
| Transport | Feature | Crate | Notes |
|---|---|---|---|
AsyncWsTransport |
tokio-tungstenite |
tokio-tungstenite | Fully async, recommended with tokio |
WsTransport |
tungstenite |
tungstenite | Sync WS bridged via spawn_blocking |
To use a fully custom transport (tower, hyper, WASM, etc.):
use chat_rs::Transport;
struct MyTransport { /* ... */ }
impl Transport for MyTransport { /* ... */ }
let client = OpenAIBuilder::new()
.with_model("gpt-4o")
.with_transport(MyTransport::new())
.build();Transport implementations live in core/src/transport/impls/. See core/AGENTS.md for the Transport trait definition.
chat-rs (root) ← Re-exports + feature flags
├── core/ ← Traits, types, Chat engine, builder, Transport trait + impls
├── providers/
│ ├── gemini/ ← Google Gemini provider
│ ├── claude/ ← Anthropic Claude provider
│ ├── openai/ ← OpenAI Responses API provider
│ └── router/ ← Multi-provider router
└── examples/
├── gemini/ ← Gemini examples
├── claude/ ← Claude examples
├── openai/ ← OpenAI examples
└── router/ ← Router strategy examples
See core/AGENTS.md and providers/AGENTS.md for detailed architecture documentation.
Run examples with the appropriate feature flags:
# Gemini
cargo run --example gemini-tools --features gemini
cargo run --example gemini-structured --features gemini
cargo run --example gemini-stream --features gemini,stream
cargo run --example gemini-embeddings --features gemini
cargo run --example gemini-code-execution --features gemini
cargo run --example gemini-google-maps --features gemini
cargo run --example gemini-image-understanding --features gemini
cargo run --example gemini-hitl --features gemini,stream
# Claude
cargo run --example claude-completion --features claude
cargo run --example claude-stream --features claude,stream
cargo run --example claude-hitl --features claude,stream
# OpenAI
cargo run --example openai-completion --features openai
cargo run --example openai-stream --features openai,stream
cargo run --example openai-structured --features openai
cargo run --example openai-embeddings --features openai
cargo run --example openai-hitl --features openai,stream
cargo run --example openai-websocket --features openai,stream,tokio-tungstenite
# Router
cargo run --example router-keyword --features router,gemini,claude
cargo run --example router-embeddings --features router,gemini,claude
cargo run --example router-capability --features router,gemini,claude
cargo run --example router-stream --features router,gemini,claude,stream
# Retry strategies
cargo run --example retry --features geminiRust 1.94 or later (edition 2024).
MIT