Skip to content

feat(anthropic): Add prompt caching support#2

Open
CeciliaZ030 wants to merge 1 commit intoBlockRunAI:mainfrom
aomi-labs:feat/prompt-caching
Open

feat(anthropic): Add prompt caching support#2
CeciliaZ030 wants to merge 1 commit intoBlockRunAI:mainfrom
aomi-labs:feat/prompt-caching

Conversation

@CeciliaZ030
Copy link

Implements Anthropic's prompt caching feature which can reduce costs by up to 90% and latency by up to 85% for repeated prompt prefixes.

Changes:

  • Add CacheControl enum with Ephemeral variant
  • Add cache_control field to SystemContent, Content, and ToolDefinition
  • Add builder methods: .with_cache_control() and .cached()
  • Support caching for system prompts, messages, and tool definitions
  • Track cache usage in Usage struct (cache_read_input_tokens, cache_creation_input_tokens)

Usage:

// Cache system prompt
let system = SystemContent::text("You are helpful.").cached();

// Cache tool definitions
let tool = ToolDefinition::new("search", "Search the web", schema).cached();

// Cache message content
let content = Content::text("Long context...").cached();

Ref: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching

Implements Anthropic's prompt caching feature which can reduce costs by up to 90%
and latency by up to 85% for repeated prompt prefixes.

Changes:
- Add CacheControl enum with Ephemeral variant
- Add cache_control field to SystemContent, Content, and ToolDefinition
- Add builder methods: .with_cache_control() and .cached()
- Support caching for system prompts, messages, and tool definitions
- Track cache usage in Usage struct (cache_read_input_tokens, cache_creation_input_tokens)

Usage:
```rust
// Cache system prompt
let system = SystemContent::text("You are helpful.").cached();

// Cache tool definitions
let tool = ToolDefinition::new("search", "Search the web", schema).cached();

// Cache message content
let content = Content::text("Long context...").cached();
```

Ref: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants