Skip to content

[Feature] Ollama and local embedding provider support #25

@Siddhant-K-code

Description

@Siddhant-K-code

Summary

Add support for Ollama and other local embedding models so users can run Distill without sending data to external APIs.

Motivation

  • Privacy-sensitive users and enterprises can't send data to OpenAI
  • Local models eliminate API costs entirely
  • Ollama is the most popular local model runner (100K+ GitHub stars)
  • Positions Distill as a fully self-contained, air-gapped solution

Embedding providers to add

Provider Interface Priority
Ollama HTTP (/api/embeddings) P0
HuggingFace TEI HTTP (OpenAI-compatible) P1
Sentence Transformers Python subprocess P2

Implementation

// pkg/embedding/ollama/client.go
type Config struct {
    Host  string // default: http://localhost:11434
    Model string // default: nomic-embed-text
}

type Client struct { ... }

func (c *Client) Embed(ctx context.Context, text string) ([]float32, error)
func (c *Client) EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)

CLI usage

# Use Ollama for embeddings
distill api --embedding-provider ollama --embedding-model nomic-embed-text

# Use with config file
# distill.yaml
embedding:
  provider: "ollama"
  model: "nomic-embed-text"
  host: "http://localhost:11434"

Config file support

Update pkg/config/config.go to support provider field in embedding config.

Deliverables

  • pkg/embedding/ollama/client.go - Ollama HTTP client
  • pkg/embedding/ollama/client_test.go - Unit tests (mock HTTP)
  • pkg/embedding/provider.go - Provider interface + factory function
  • Update cmd/api.go and cmd/serve.go to accept --embedding-provider flag
  • Update pkg/config/config.go with provider field
  • Update README with Ollama setup instructions
  • docker-compose.yml example with Ollama sidecar

Acceptance Criteria

  • distill api --embedding-provider ollama works with local Ollama
  • No data leaves the machine
  • Batch embedding support
  • Graceful error when Ollama is not running
  • Config file supports provider selection

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions