Skip to content

Add PostHog LLM analytics with tags, session_ids, and host_ids#28

Open
ufkhan97 wants to merge 2 commits intomasterfrom
feat/posthog-llm-tagging
Open

Add PostHog LLM analytics with tags, session_ids, and host_ids#28
ufkhan97 wants to merge 2 commits intomasterfrom
feat/posthog-llm-tagging

Conversation

@ufkhan97
Copy link
Collaborator

@ufkhan97 ufkhan97 commented Dec 10, 2025

Summary

  • Adds tag property to PostHog $ai_generation events to identify different LLM use cases (e.g., 'chat', 'summary_generation',
    'cross_pollination_question')
  • Adds session_ids and host_ids arrays to enable cost attribution per session and per host
    • session_ids: The host_sessions.id identifying the workshop
    • host_ids: The owner's user_id from permissions table (Auth0 sub)
  • Creates helper functions getSessionOwner() and getSessionOwners() to fetch owner from permissions table

Changes

Analytics enhancement:

  • Extended LLM.chat() interface with tag, sessionIds, and hostIds parameters
  • All LLM call sites now pass contextual information for tracking

Files updated:

  • src/lib/modelConfig.ts - Core LLM wrapper with PostHog capture
  • src/lib/db.ts - New session owner lookup helpers
  • src/app/api/llamaUtils.ts - Chat handler
  • src/lib/sessionGenerator.ts - Session simulation
  • src/lib/characterGenerator.ts - Character generation
  • src/lib/crossPollinationManager.ts - Cross-pollination logic
  • src/lib/monica/monicaMultiSession.ts - Multi-session AI
  • src/lib/monica/monicaSingleSessionRAG.ts - Single-session RAG
  • src/lib/summaryMultiSession.ts - Summary generation

Test plan

  • Verify PostHog events include tag, session_ids, and host_ids properties
  • Test chat functionality to confirm LLM calls still work correctly
  • Validate cost attribution queries work with new properties

EDIT: Note on package-lock.json:

This PR includes ~870 lines of deletions in package-lock.json. These are optional peer dependencies (sharp image processing binaries, @emnapi/runtime, etc.) that got removed when npm resolved the lock file on my machine. Likely due to different npm version or platform.

These are platform-specific optional packages that get installed as-needed. It shouldn't affect functionality, but if you'd prefer to keep the lock file unchanged, I can revert just that file from the PR.

ufkhan97 and others added 2 commits December 10, 2025 13:06
- Add tag parameter to LLM.chat() method signature
- Tag all LLM calls to identify their use case:
  - chat: participant chat interactions
  - monica_ai: Monica AI multi-session queries
  - monica_ai_rag: Monica AI single-session RAG queries
  - summary_generation: session summary generation
  - session_completion_check: checking if session is complete
  - session_user_response: simulated user responses
  - builder_create_prompt, builder_edit_prompt, builder_summary_prompt
  - cross_pollination_analysis, cross_pollination_question
  - data_export, file_analysis, query_type_analysis
  - simulated_participant, form_answer_generation, character_generation
- Resolve merge conflict in modelConfig.ts

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Enables cost attribution per session and host by adding session_ids
and host_ids arrays to $ai_generation events.

- session_ids: The host_sessions.id identifying the workshop
- host_ids: The owner's user_id from permissions table (Auth0 sub)

Changes:
- Add sessionIds/hostIds params to LLM.chat() interface
- Create getSessionOwner() and getSessionOwners() helpers in db.ts
- Update all LLM call sites to pass session/host context:
  - llamaUtils.ts (chat handler)
  - sessionGenerator.ts
  - characterGenerator.ts
  - crossPollinationManager.ts
  - monicaMultiSession.ts
  - monicaSingleSessionRAG.ts
  - summaryMultiSession.ts

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@vercel
Copy link

vercel bot commented Dec 10, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
harmonica-web-app Ready Ready Preview Comment Dec 10, 2025 6:29pm
harmonica-web-app-dev Ready Ready Preview Comment Dec 10, 2025 6:29pm

@ufkhan97
Copy link
Collaborator Author

Note on package-lock.json: This PR includes ~870 lines of deletions in package-lock.json. These are optional peer dependencies (sharp image processing binaries, @emnapi/runtime, etc.) that got removed when npm resolved the lock file on my machine. Likely due to different npm version or platform.

These are platform-specific optional packages that get installed as-needed. It shouldn't affect functionality, but if you'd prefer to keep the lock file unchanged, I can revert just that file from the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant