Add local inference service for task summarization

[vanpelt]

Summary

Adds local GGUF model inference using llama.cpp via yzma, enabling on-device task summarization and git branch name generation with our fine-tuned Gemma 3 270M model.

Key Features

• CLI Commands

  • catnip summarize "task description" - Generate task summary and branch name
  • catnip download - Pre-download model and llama.cpp libraries

• REST API

  • POST /v1/inference/summarize - Inference endpoint for programmatic access
  • GET /v1/inference/status - Check inference service availability

• Auto-downloading

  • Models cached to ~/.catnip/models/
  • llama.cpp libraries auto-downloaded for current platform to ~/.catnip/lib/

Critical Bug Fix

Fixed inference producing incorrect outputs (always returning "Add Dark Mode" from examples instead of actual summaries).

Root cause: Missing BOS (Beginning of Sequence) token when tokenizing prompts for Gemma models.

Fix: Set addSpecial=true in tokenization call to include required special tokens.

Test plan

• Verify catnip summarize produces varied, contextually appropriate outputs
• Compare output quality with Ollama using same model
• Test multiple prompts to confirm no example contamination
• Verify lint passes

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 1.23839% with 638 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
container/internal/services/downloader.go	0.00%	235 Missing ⚠️
container/internal/services/inference.go	0.00%	201 Missing ⚠️
container/internal/cmd/download.go	6.57%	71 Missing ⚠️
container/internal/handlers/inference.go	0.00%	52 Missing ⚠️
container/internal/cmd/summarize.go	9.09%	30 Missing ⚠️
container/internal/services/stderr_unix.go	0.00%	28 Missing ⚠️
container/internal/cmd/serve.go	0.00%	17 Missing ⚠️
container/internal/tui/initialization_commands.go	0.00%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	golang/​github.com/​hybridgroup/​yzma@​v0.9.0

View full report

[socket-security]

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action	Severity	Alert  (click "▶" to expand/collapse)
Warn		Native binaries present: golang github.com/jupiterrider/ffi Location: Package overview From: container/go.mod → golang/github.com/hybridgroup/yzma@v0.9.0 → golang/github.com/jupiterrider/ffi@v0.5.1 ℹ Read more on: This package | This alert | Why is native code a concern? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Verify that the inclusion of native code is expected and necessary for this package's functionality. If it is unnecessary or unexpected, consider using alternative packages without native code to mitigate potential risks. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore golang/github.com/jupiterrider/ffi@v0.5.1. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report