fix: message delta visibility for anthropic fixed #2212
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🦋 Changeset detectedLatest commit: 92d840a The changes in this PR will be included in the next version bump. This PR includes changesets to release 13 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
| let lastChunk: ChatResponseChunk | undefined; | ||
| const toolCalls: Map<string, AgentToolCall> = new Map(); | ||
| for await (const chunk of responseStream) { | ||
| lastChunk = chunk; |
There was a problem hiding this comment.
Why is this needed?
There was a problem hiding this comment.
because in the following code snippet
return {
response: message,
toolCalls: Array.from(toolCalls.values()),
raw: lastChunk?.raw,
currentAgentName: this.name,
};
the raw chunk that is being returned takes the value from lastChunk.raw, in the earlier code, lastChunk was beind defined, but never assigned a value, thus lastChunk always had the value as undefined, so thats why i assigned it tha lastChunk that we see in the loop, sometime it could even be empty text, but even if its empty, it still sends metadata
| "@llamaindex/anthropic": patch | ||
| --- | ||
|
|
||
| fix: message delta visibility for anthropic |
There was a problem hiding this comment.
We need a change set what we are fixing for workflow in this PR
There was a problem hiding this comment.
Oh I see, I was going through CONTRIBUTING.md and since this wasn't a feature, I thought that this wouldn't need a changeset, I'm sorry about it
There was a problem hiding this comment.
yes. sure, that would be great!
@llamaindex/autotool
@llamaindex/community
@llamaindex/core
@llamaindex/env
@llamaindex/experimental
llamaindex
@llamaindex/node-parser
@llamaindex/readers
@llamaindex/tools
@llamaindex/wasm-tools
@llamaindex/workflow
@llamaindex/anthropic
@llamaindex/assemblyai
@llamaindex/aws
@llamaindex/clip
@llamaindex/cohere
@llamaindex/deepinfra
@llamaindex/deepseek
@llamaindex/discord
@llamaindex/excel
@llamaindex/fireworks
@llamaindex/google
@llamaindex/groq
@llamaindex/huggingface
@llamaindex/jinaai
@llamaindex/mistral
@llamaindex/mixedbread
@llamaindex/notion
@llamaindex/ollama
@llamaindex/openai
@llamaindex/perplexity
@llamaindex/portkey-ai
@llamaindex/replicate
@llamaindex/together
@llamaindex/vercel
@llamaindex/vllm
@llamaindex/voyage-ai
@llamaindex/xai
@llamaindex/astra
@llamaindex/azure
@llamaindex/chroma
@llamaindex/elastic-search
@llamaindex/firestore
@llamaindex/milvus
@llamaindex/mongodb
@llamaindex/pinecone
@llamaindex/postgres
@llamaindex/qdrant
@llamaindex/supabase
@llamaindex/upstash
@llamaindex/weaviate
commit: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR aims to fix the message_delta not taken into consideration for Anthropic's LLMs as indicate in issue #2207
It solves the following issues
message_deltais emiting now, so the usage meta data isn't dropped and is actually reaching the userundefined, this is fixed as the lastChunk is being assigned the value of the chunk that is read last in the loop, even though it might have empty content, their is usage data which can be passed on.rawas one of the fields in AgentWorkflow, so callers ofagent.runrecieve the usage when it is sent by thetakeStepfunctionThe following tests were ran and they were passing for the same
pnpm --filter @llamaindex/workflow testpnpm --filter @llamaindex/anthropic test