-
-
Notifications
You must be signed in to change notification settings - Fork 14
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Summary
Add support for Ollama and other local embedding models so users can run Distill without sending data to external APIs.
Motivation
- Privacy-sensitive users and enterprises can't send data to OpenAI
- Local models eliminate API costs entirely
- Ollama is the most popular local model runner (100K+ GitHub stars)
- Positions Distill as a fully self-contained, air-gapped solution
Embedding providers to add
| Provider | Interface | Priority |
|---|---|---|
| Ollama | HTTP (/api/embeddings) |
P0 |
| HuggingFace TEI | HTTP (OpenAI-compatible) | P1 |
| Sentence Transformers | Python subprocess | P2 |
Implementation
// pkg/embedding/ollama/client.go
type Config struct {
Host string // default: http://localhost:11434
Model string // default: nomic-embed-text
}
type Client struct { ... }
func (c *Client) Embed(ctx context.Context, text string) ([]float32, error)
func (c *Client) EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)CLI usage
# Use Ollama for embeddings
distill api --embedding-provider ollama --embedding-model nomic-embed-text
# Use with config file
# distill.yaml
embedding:
provider: "ollama"
model: "nomic-embed-text"
host: "http://localhost:11434"Config file support
Update pkg/config/config.go to support provider field in embedding config.
Deliverables
-
pkg/embedding/ollama/client.go- Ollama HTTP client -
pkg/embedding/ollama/client_test.go- Unit tests (mock HTTP) -
pkg/embedding/provider.go- Provider interface + factory function - Update
cmd/api.goandcmd/serve.goto accept--embedding-providerflag - Update
pkg/config/config.gowith provider field - Update README with Ollama setup instructions
-
docker-compose.ymlexample with Ollama sidecar
Acceptance Criteria
-
distill api --embedding-provider ollamaworks with local Ollama - No data leaves the machine
- Batch embedding support
- Graceful error when Ollama is not running
- Config file supports provider selection
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request