Skip to content

Add L.I.S.A. LiveKit agent with web UI and enhanced widgets#8

Open
breakingcircuits1337 wants to merge 9 commits intoNlouis38:mainfrom
breakingcircuits1337:claude/update-readme-latest-1YNUl
Open

Add L.I.S.A. LiveKit agent with web UI and enhanced widgets#8
breakingcircuits1337 wants to merge 9 commits intoNlouis38:mainfrom
breakingcircuits1337:claude/update-readme-latest-1YNUl

Conversation

@breakingcircuits1337
Copy link
Copy Markdown

Summary

This PR introduces L.I.S.A. (Life Integrated System Architecture), a new LiveKit-based voice agent deployment with a cyberpunk-themed web frontend, alongside significant enhancements to the existing ADA system. The changes include a complete web UI, LiveKit agent implementation, new widget capabilities (calendar, email), and improved system architecture.

Key Changes

New LiveKit Agent & Web Infrastructure

  • ADA/ADA_LiveKit.py: New LiveKit agent implementation with STT/TTS/LLM plugin support, function tools for system info, timers, and todo management
  • server.py: FastAPI backend serving token generation for LiveKit connections and static frontend files
  • frontend/index.html, frontend/app.js, frontend/styles.css: Complete cyberpunk-themed web UI with:
    • Real-time audio/video streaming via LiveKit WebRTC
    • Hand gesture recognition using MediaPipe
    • System stats visualization (CPU/RAM bars)
    • Chat interface with message history
    • Animated holographic visualizer with pulsing core and rotating rings
    • Sci-fi aesthetic with neon colors, glitch effects, and scanline overlays

New Widgets & Enhanced Functionality

  • ADA/WIDGETS/calendar_widget.py: Calendar event management (add, list, delete events with date/time validation)
  • ADA/WIDGETS/email_client.py: Email capabilities (send via SMTP, read via IMAP)
  • ADA/WIDGETS/to_do_list.py: Refactored to use JSON persistence instead of in-memory storage
  • ADA/WIDGETS/system.py: Refactored to return formatted string instead of printing; added return type hints

Docker & Deployment

  • docker-compose.yml: LiveKit server + Redis stack for local development
  • livekit.yaml: LiveKit server configuration with dev credentials
  • start_all.bat: One-click startup script for Windows (Docker + API server)
  • requirements_livekit.txt: Dependencies for LiveKit agent deployment

Configuration & Documentation

  • .env.local_example: Environment template for LiveKit, API keys, and email credentials
  • README.md: Comprehensive update documenting all three ADA versions, project structure, setup instructions, and feature matrix

Integration Updates

  • ADA/ADA_Local.py: Added tool definitions and available_tools mapping for calendar, email, and todo functions; improved tool call extraction with proper argument parsing

Notable Implementation Details

  • VAD Fix: Custom FixedSileroVAD wrapper in ADA_LiveKit.py addresses missing super().__init__() call in livekit-plugins-silero
  • Gesture Recognition: Frontend integrates MediaPipe Hands for real-time hand tracking and gesture detection
  • Persistent Storage: Calendar and todo widgets use JSON files for data persistence
  • Email Integration: Supports configurable SMTP/IMAP servers (defaults to Gmail)
  • Responsive UI: CSS Grid layout with glassmorphism effects, backdrop blur, and neon glow animations
  • Tool Execution: ADA_Local now safely executes tool calls with proper error handling and argument validation

https://claude.ai/code/session_01MKgav2A2hL7jgQi4XovPXJ

Gemini Bot and others added 9 commits January 22, 2026 03:50
- Add LiveKit / L.I.S.A. agent section with setup and run instructions
- Document web frontend (frontend/ directory) and FastAPI server
- Add calendar_widget and email_client to integrated widgets table
- Fix to_do_list note — it is now integrated in ada_local and L.I.S.A.
- Document docker-compose / livekit.yaml / start_all.bat for LiveKit stack
- Add requirements_livekit.txt install path
- Update .env section to cover all new keys (LiveKit, Deepgram, OpenAI, email)
- Add full project structure tree
- Update feature list and widget availability matrix across all versions

https://claude.ai/code/session_01MKgav2A2hL7jgQi4XovPXJ
New widget: ADA/WIDGETS/network_security.py
- network.get_interfaces()         — list interfaces with IP/MAC/status
- network.arp_table()              — read OS ARP cache (no packets sent)
- network.active_connections()     — list TCP/UDP connections with state filter
- network.check_suspicious_connections() — flag unusual ports and outbound public IPs
- network.scan_lan()               — ARP scan via scapy or ping sweep fallback
- network.port_scan()              — TCP port check with common/range/list modes
- network.ping()                   — ICMP ping with latency report
- network.dns_lookup()             — forward + reverse DNS resolution

Wired into ADA_Local.py: import, instruction prompt descriptions, available_tools entries.
Updated README: widget table, network security section with example commands, scapy install note.

https://claude.ai/code/session_01MKgav2A2hL7jgQi4XovPXJ
calendar_widget.py:
- Backed by Google Calendar API (OAuth 2.0 Desktop flow)
- add_event() creates events with popup + email reminder (default 30 min)
- list_events() pulls upcoming events from Google Calendar (or filtered by date)
- delete_event() finds and deletes by title from upcoming events
- Falls back with helpful error if google libs or credentials.json are missing
- Auto-detects local timezone via tzlocal (falls back to UTC)
- Token cached in calendar_token.json for subsequent runs

ADA_Local.py:
- Updated calendar.add prompt description to include reminder_minutes param
- Instructs ADA to always use calendar.add when user says "remind me"

.env.local_example + README:
- Added GOOGLE_CALENDAR_CREDENTIALS_FILE env var
- Added step-by-step Google Calendar OAuth setup guide
- Added required pip packages for calendar integration

https://claude.ai/code/session_01MKgav2A2hL7jgQi4XovPXJ
…llama)

ADA_Local.py:
- Added _chat_stream(messages) generator — dispatches to the right provider,
  yields plain string chunks so send_prompt stays provider-agnostic
- Replaced both ollama.chat() call sites with for chunk_content in self._chat_stream()
- Provider init in __init__ reads ADA_LLM_PROVIDER env var (default: ollama)
  - ollama   : existing behaviour unchanged, model overrideable via OLLAMA_MODEL
  - claude   : Anthropic SDK, system msg extracted automatically, streams text_stream
  - azure-mistral : OpenAI-compatible client pointed at Azure Foundry endpoint
  - azure-kimi    : same pattern, separate endpoint for Kimi 2.5 thinking model
- Missing library → warning printed, graceful fallback to Ollama
- Ollama import and default model untouched

.env.local_example + README:
- Added ADA_LLM_PROVIDER, ANTHROPIC_*, AZURE_FOUNDRY_API_KEY, AZURE_*_ENDPOINT/MODEL
- Added LLM Providers table with install notes and Azure endpoint lookup instructions

https://claude.ai/code/session_01MKgav2A2hL7jgQi4XovPXJ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants