Skip to content

Conversation

@balloob
Copy link
Member

@balloob balloob commented Oct 27, 2025

Breaking change

Proposed change

To improve debugging of LLMs, adding a set of endpoints to allow voice assistant debug sessions to look inside the chat log data to see what was sent between LLM and HA.

Frontend: home-assistant/frontend#27678

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:
  • Link to developer documentation pull request:
  • Link to frontend pull request:

Checklist

  • I understand the code I am submitting and can explain how it works.
  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • I have followed the perfect PR recommendations
  • The code has been formatted using Ruff (ruff format homeassistant tests)
  • Tests have been added to verify that the new code works.
  • Any generated code has been carefully reviewed for correctness and compliance with project standards.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

To help with the load of incoming pull requests:

@home-assistant
Copy link

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (homeassistant) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of homeassistant can trigger bot actions by commenting:

  • @home-assistant close Closes the pull request.
  • @home-assistant rename Awesome new title Renames the pull request.
  • @home-assistant reopen Reopen the pull request.
  • @home-assistant unassign homeassistant Removes the current integration label and assignees on the pull request, add the integration domain after the command.
  • @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
  • @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

@home-assistant
Copy link

Hey there @home-assistant/core, @synesthesiam, @arturpragacz, mind taking a look at this pull request as it has been labeled with an integration (conversation) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of conversation can trigger bot actions by commenting:

  • @home-assistant close Closes the pull request.
  • @home-assistant rename Awesome new title Renames the pull request.
  • @home-assistant reopen Reopen the pull request.
  • @home-assistant unassign conversation Removes the current integration label and assignees on the pull request, add the integration domain after the command.
  • @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
  • @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds WebSocket subscription endpoints for chat log monitoring to improve debugging of LLM interactions in voice assistant sessions. The implementation allows real-time observation of conversation data exchanged between LLMs and Home Assistant.

Key changes:

  • Added two WebSocket subscription endpoints for chat log access: individual conversation and index (all conversations)
  • Implemented event notification system for chat log lifecycle (created, updated, deleted, content_added)
  • Added as_dict() serialization methods and timestamp tracking to content models

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
homeassistant/components/conversation/http.py Implements WebSocket subscription handlers for chat logs
homeassistant/components/conversation/chat_log.py Adds event subscription infrastructure and serialization methods
homeassistant/components/conversation/const.py Defines chat log event type enumeration
homeassistant/components/homeassistant/const.py Fixes typo in exposed entities key name
tests/components/conversation/test_http.py Tests WebSocket subscription endpoints and event flows
tests/components/conversation/test_chat_log.py Tests chat log subscription and event notification system

@arturpragacz
Copy link
Contributor

I'm not a fan of the fact that this adds yet another mechanism to report events related to voice.

@balloob
Copy link
Member Author

balloob commented Oct 31, 2025

@arturpragacz I hear you, however, chat log is not voice specific. It's the way we track the history of an interaction with an AI from either conversation or AI task. If we were to add these events into the Assist Pipeline websocket endpoint, we would now need to also make that one aware of AI task. Would you have another suggestion?

@arturpragacz
Copy link
Contributor

I hear you, however, chat log is not voice specific. It's the way we track the history of an interaction with an AI from either conversation or AI task. If we were to add these events into the Assist Pipeline websocket endpoint, we would now need to also make that one aware of AI task. Would you have another suggestion?

Can we not have a way to register a callback in the chat log, which can then be used in the higher layer. So the assist pipeline would use it to provide those events through its websocket and the AI task could do what is the best for it. I don't think we have any debug interface for AI tasks at this point anyway, unless I'm missing something.

@balloob
Copy link
Member Author

balloob commented Nov 8, 2025

This is the debugging endpoint for AI tasks.

@frenck frenck added this to the 2025.12.0b0 milestone Nov 13, 2025
Copy link
Contributor

@arturpragacz arturpragacz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the open comments by Copilot.

balloob and others added 4 commits November 24, 2025 16:04
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@balloob balloob force-pushed the chat-log-subscription branch from 758d5ca to e0ebc78 Compare November 24, 2025 21:04
Copy link
Contributor

@arturpragacz arturpragacz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, let's merge it now, but investigate how to include conversation trace in this.

@arturpragacz arturpragacz merged commit e780e3d into dev Nov 24, 2025
62 checks passed
@arturpragacz arturpragacz deleted the chat-log-subscription branch November 24, 2025 21:31
@github-actions github-actions bot locked and limited conversation to collaborators Nov 25, 2025
@frenck frenck removed this from the 2025.12.0b0 milestone Nov 26, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants