Skip to content

fix(google): capture usage_metadata before early continues in streaming#5404

Open
Panmax wants to merge 2 commits intolivekit:mainfrom
Panmax:fix/google-llm-streaming-usage-metadata
Open

fix(google): capture usage_metadata before early continues in streaming#5404
Panmax wants to merge 2 commits intolivekit:mainfrom
Panmax:fix/google-llm-streaming-usage-metadata

Conversation

@Panmax
Copy link
Copy Markdown
Contributor

@Panmax Panmax commented Apr 10, 2026

Summary

  • Fix prompt_tokens: 0, completion_tokens: 0 in LLM metrics for Gemini streaming requests
  • Move the usage_metadata check before the two continue statements in LLMStream._run so token counts are always captured
  • In Gemini's streaming API, the final chunk typically carries usage_metadata with complete token counts but has empty candidates or content.parts, causing the existing code to skip the usage check entirely

Details

In livekit-plugins-google/livekit/plugins/google/llm.py, the streaming loop has two early continue statements:

  1. if not response.candidates: continue
  2. if not candidate.content or not candidate.content.parts: continue

Both skip past the usage_metadata check at the end of the loop body. Since Gemini's final streaming chunk often has a finish_reason but empty content, the usage data (which is only fully populated on this last chunk) is never captured.

The fix moves the usage_metadata check to immediately after the prompt_feedback check, before any continue statements.

Test plan

  • Existing tests pass
  • Lint and type checks pass
  • Manual verification: run a Gemini streaming session and confirm LLM metrics show non-zero prompt_tokens, completion_tokens, and tokens_per_second

🤖 Generated with Claude Code

…streaming

In Gemini's streaming API, the final chunk often has usage_metadata with
token counts but empty candidates or content.parts. Two early `continue`
statements caused the usage_metadata check to be skipped, resulting in
prompt_tokens=0, completion_tokens=0 in LLM metrics.

Move the usage_metadata check to before any continue statements so it is
always evaluated regardless of whether the chunk has content.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 2 additional findings.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant