Skip to content

ShipItAndPray/turboquant

BranchesTags

Repository files navigation

 ████████╗██╗   ██╗██████╗ ██████╗  ██████╗  ██████╗ ██╗   ██╗ █████╗ ███╗   ██╗████████╗
 ╚══██╔══╝██║   ██║██╔══██╗██╔══██╗██╔═══██╗██╔═══██╗██║   ██║██╔══██╗████╗  ██║╚══██╔══╝
    ██║   ██║   ██║██████╔╝██████╔╝██║   ██║██║   ██║██║   ██║███████║██╔██╗ ██║   ██║
    ██║   ██║   ██║██╔══██╗██╔══██╗██║   ██║██║▄▄ ██║██║   ██║██╔══██║██║╚██╗██║   ██║
    ██║   ╚██████╔╝██║  ██║██████╔╝╚██████╔╝╚██████╔╝╚██████╔╝██║  ██║██║ ╚████║   ██║
    ╚═╝    ╚═════╝ ╚═╝  ╚═╝╚═════╝  ╚═════╝  ╚══▀▀═╝  ╚═════╝ ╚═╝  ╚═╝╚═╝  ╚═══╝   ╚═╝

6x Compression for Vectors, Embeddings, and LLMs

Based on Google's TurboQuant — PolarQuant + QJL. No training required. Near-zero accuracy loss.

24 Adapters • Quick Start • Benchmarks • How It Works • LLM CLI • Platforms • CI/CD

Adapters — Plug and Play for 24 Systems

3 lines of code. No forks, no patches, no recompilation. Wrap your existing client, get 6x compression.

from turboquant.core import TurboQuantEncoder
from turboquant.adapters.redis import RedisTurboCache

encoder = TurboQuantEncoder(dim=768)
cache = RedisTurboCache(encoder, your_existing_redis_client)
cache.put("doc:1", embedding)  # 3KB → 500 bytes

Every adapter has the same API: put · get · search · put_batch · get_batch · delete · stats

Caches

Adapter Install Key Feature
Redis pip install redis Pipeline batching, SCAN search, TTL, key prefixing
Memcached pip install pymemcache get_multi/set_multi, CAS atomic updates
Ehcache pip install py4j Java JVM bridge (Py4J) or REST API, Ehcache 2 & 3
Hazelcast pip install hazelcast-python-client Distributed cluster, put_all/get_all

Databases

Adapter Install Key Feature
PostgreSQL pip install psycopg2-binary BYTEA + optional pgvector hybrid search, JSONB metadata
MySQL pip install mysql-connector-python MEDIUMBLOB storage, executemany bulk insert
SQLite (built-in — zero deps) WAL mode, JSON1 metadata, great for local dev
MongoDB pip install pymongo BSON Binary, Atlas Vector Search aggregation pipeline
DynamoDB pip install boto3 Binary attribute, batch_write_item (25/batch), TTL
Cassandra pip install cassandra-driver Prepared statements, UNLOGGED BATCH, native TTL

Vector Databases

Adapter Install Key Feature
Pinecone pip install pinecone-client Native ANN + TurboQuant reranking for higher recall
Qdrant pip install qdrant-client HNSW search + rerank, payload filtering
ChromaDB pip install chromadb Local/server mode, metadata where-filtering
Milvus pip install pymilvus IVF/HNSW index + TurboQuant rerank
Weaviate pip install weaviate-client Schema-based, near_vector + rerank
FAISS pip install faiss-cpu Local ANN index, save/load to disk, rerank mode

Search Engines

Adapter Install Key Feature
Elasticsearch pip install elasticsearch Binary field + dense_vector kNN, bulk API
OpenSearch pip install opensearch-py k-NN plugin (nmslib/faiss engine), compressed-only mode

Object Storage

Adapter Install Key Feature
AWS S3 pip install boto3 ~500B objects, concurrent ThreadPool upload
Google Cloud Storage pip install google-cloud-storage Blob metadata, concurrent upload
Azure Blob pip install azure-storage-blob Container-based, blob metadata

Embedded Key-Value Stores

Adapter Install Key Feature
LMDB pip install lmdb Memory-mapped B+ tree, zero-copy reads, ACID
RocksDB pip install python-rocksdb WriteBatch, LSM-tree (less write amplification with smaller values)

Streaming

Adapter Install Key Feature
Apache Kafka pip install confluent-kafka Producer + Consumer, 6x smaller messages, metadata support

Full adapter docs with examples: adapters/README.md

Quick Start

Install

pip install numpy   # Only dependency for core engine

# Then install your backend's client:
pip install redis               # for Redis adapter
pip install psycopg2-binary     # for PostgreSQL adapter
pip install pymongo             # for MongoDB adapter
# ... etc

Compress and Store Vectors

from turboquant.core import TurboQuantEncoder, TurboQuantConfig

# Create encoder (reuse across your app)
config = TurboQuantConfig(bits=4, block_size=32, qjl_proj_dim=64)
encoder = TurboQuantEncoder(dim=768, config=config)

# Compress a single vector
import numpy as np
vector = np.random.randn(768).astype(np.float32)
compressed = encoder.encode(vector)

print(f"Original:   {768 * 4} bytes")
print(f"Compressed: {compressed.nbytes()} bytes")
print(f"Ratio:      {compressed.compression_ratio():.1f}x")

# Decompress
reconstructed = encoder.decode(compressed)

# Serialize for any storage
raw_bytes = compressed.to_bytes()
restored = type(compressed).from_bytes(raw_bytes)

Use with Any Backend

# Redis
import redis
from turboquant.adapters.redis import RedisTurboCache
cache = RedisTurboCache(encoder, redis.Redis(), prefix="emb:", ttl=3600)

# PostgreSQL with pgvector
from turboquant.adapters.postgresql import PostgresTurboCache
cache = PostgresTurboCache(encoder, dsn="postgresql://localhost/mydb", use_pgvector=True)

# MongoDB with Atlas Vector Search
from pymongo import MongoClient
from turboquant.adapters.mongodb import MongoTurboCache
cache = MongoTurboCache(encoder, MongoClient(), db="myapp", collection="embeddings")

# S3
from turboquant.adapters.s3 import S3TurboCache
cache = S3TurboCache(encoder, bucket="my-vectors", prefix="embeddings/")

# SQLite (zero deps)
from turboquant.adapters.sqlite import SQLiteTurboCache
cache = SQLiteTurboCache(encoder, db_path="vectors.db")

# All adapters — same API:
cache.put("doc:1", vector)
cache.put_batch({"doc:2": v2, "doc:3": v3})
vec = cache.get("doc:1")
results = cache.search(query_vector, k=10)
print(cache.stats())

Vector DB Reranking

For Pinecone, Qdrant, Milvus, etc. — use native ANN for candidates, TurboQuant for precision:

from turboquant.adapters.qdrant import QdrantTurboCache
cache = QdrantTurboCache(encoder, qdrant_client, collection="docs")

results = cache.search(query, k=10, mode="rerank")      # ANN + TQ rerank (best quality)
results = cache.search(query, k=10, mode="native")      # ANN only (fastest)
results = cache.search(query, k=10, mode="compressed")   # TQ only (no ANN index needed)

Build Your Own Adapter

Subclass BaseTurboAdapter — implement 4 methods, get the full API for free:

from turboquant.adapters._base import BaseTurboAdapter

class MyCache(BaseTurboAdapter):
    def _raw_get(self, key): ...       # return bytes or None
    def _raw_set(self, key, value): ...  # store bytes
    def _raw_delete(self, key): ...    # return bool
    def _raw_keys(self, pattern): ...  # return list of keys

# You now have: put, get, search, put_batch, get_batch, delete, stats

Compression Benchmarks

4-bit quantization, block_size=32, QJL proj_dim=64:

Dimension Compression Cosine Similarity Bytes per Vector
128 5.5x 0.990 94
384 6.1x 0.973 254
768 6.2x 0.949 494
1536 6.3x 0.907 974

Memory Savings at Scale

Scenario Raw float32 TurboQuant Saved
10K vectors, dim=128 5 MB 940 KB 82%
100K vectors, dim=384 154 MB 25 MB 83%
1M vectors, dim=768 3.1 GB 494 MB 84%
10M vectors, dim=1536 61.4 GB 9.7 GB 84%

Throughput

Operation Speed (dim=768)
Encode ~1,000 vec/s
Decode ~1,800 vec/s
Similarity ~500 pairs/s

How It Works

Based on Google's TurboQuant research — two-stage compression, no training required:

Stage 1: PolarQuant

  1. Random orthogonal rotation — spreads information uniformly across all vector components
  2. Block-wise quantization — each block of 32 values gets its own scale factor, quantized to N bits
  3. Norm preservation — vector magnitude stored separately at float16 precision

Stage 2: QJL (Quantized Johnson-Lindenstrauss)

  1. Random projection of the quantization residual into a lower-dimensional space
  2. 1-bit sign quantization — each projected value becomes just +1 or -1
  3. Unbiased error correction — mathematically proven to eliminate quantization bias
Input Vector (float32)          Compressed Output (~6x smaller)
   ┌─────────────┐              ┌──────────────────────────┐
   │ [0.23, -0.1,│              │ norm (2B) + block scales  │
   │  0.45, 0.67,│   encode()   │ (N*4B) + packed N-bit    │
   │  ...768 dim ]│ ──────────→  │ values + QJL sign bits   │
   │ 3,072 bytes  │              │ ~494 bytes               │
   └─────────────┘              └──────────────────────────┘

LLM Quantization CLI

TurboQuant also includes a CLI for compressing HuggingFace LLMs to GGUF/GPTQ/AWQ:

turboquant meta-llama/Llama-3.1-8B-Instruct --format gguf --bits 4

That's it. Your 16GB model is now 4GB. Ship it to Ollama, vLLM, or llama.cpp.

pip install turboquant[all]  # Install all LLM backends

Target Platforms

Don't know which format to use? Just tell TurboQuant where you want to run it.

Ollama (one command, ready to run)

turboquant meta-llama/Llama-3.1-8B-Instruct --target ollama --bits 4

This quantizes to GGUF, auto-generates a Modelfile with the correct chat template, and tells you the exact ollama create command to run.

vLLM

turboquant meta-llama/Llama-3.1-8B-Instruct --target vllm --bits 4

Auto-selects AWQ (best GPU throughput for vLLM).

LM Studio / llama.cpp

turboquant meta-llama/Llama-3.1-8B-Instruct --target lmstudio --bits 4
turboquant meta-llama/Llama-3.1-8B-Instruct --target llamacpp --bits 4

Publish to HuggingFace

Quantize any model and publish to HuggingFace Hub in one command:

turboquant meta-llama/Llama-3.1-8B-Instruct \
  --format gguf --bits 4 \
  --push-to-hub yourname/Llama-3.1-8B-Instruct-GGUF

Requires: huggingface-cli login or HF_TOKEN environment variable.

Quality Evaluation

turboquant meta-llama/Llama-3.1-8B-Instruct --format gguf --bits 4 --eval
Perplexity Grade Meaning
< 10 EXCELLENT Minimal quality loss
10-20 GOOD Acceptable for most use cases
20-50 FAIR Some degradation, consider higher bits
> 100 POOR Model may be broken

Smart Recommendations

turboquant meta-llama/Llama-3.1-8B-Instruct --recommend

Detects your hardware (Apple Silicon, NVIDIA GPU, CPU-only) and recommends the best format + bits.

GitHub Action

CI/CD pipeline for LLM quantization. Auto-quantize after fine-tuning.

# .github/workflows/quantize.yml
name: Quantize Model
on:
  workflow_dispatch:
    inputs:
      model:
        description: 'Model to quantize'
        required: true

jobs:
  quantize:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ShipItAndPray/turboquant@master
        with:
          model: ${{ inputs.model }}
          format: gguf
          bits: 4
          eval: true
          push-to-hub: yourname/model-GGUF
          hf-token: ${{ secrets.HF_TOKEN }}

Action Inputs

Input Required Default Description
model Yes — HuggingFace model ID or local path
format No gguf gguf, gptq, awq, or all
bits No 4 2, 3, 4, 5, or 8
target No — ollama, vllm, llamacpp, lmstudio
push-to-hub No — HuggingFace repo to upload to
eval No false Run quality evaluation
hf-token No — HuggingFace API token

LLM Formats

Format Best For Engine GPU?
GGUF Local/CPU, Ollama, LM Studio llama.cpp No
GPTQ GPU serving, high throughput vLLM, TGI Yes
AWQ Fast GPU inference vLLM, TGI Yes

Don't know? Run turboquant your-model --recommend.

Supported Architectures

LLaMA (1-3.3), Mistral/Mixtral, Qwen (1.5-2.5), Phi (1-4), GPT-2/J/NeoX, Gemma, DeepSeek, and any HuggingFace model with .safetensors or .bin weights.

All CLI Options

turboquant MODEL [OPTIONS]

Positional:
  MODEL                     HuggingFace model ID or local path

Formats:
  --format, -f FORMAT       gguf, gptq, awq, or all (default: gguf)
  --bits, -b BITS           2, 3, 4, 5, or 8 (default: 4)
  --output, -o DIR          Output directory (default: ./turboquant-output)

Target Platforms:
  --target, -t TARGET       ollama, vllm, llamacpp, lmstudio

Publishing:
  --push-to-hub REPO        Upload to HuggingFace Hub (e.g. user/model-GGUF)

Quality:
  --eval                    Run perplexity evaluation after quantization
  --recommend               Show hardware-aware format recommendation

Info:
  --info                    Show model details without quantizing
  --check                   Show available backends and hardware

Requirements

  • Python 3.9+
  • NumPy (only dependency for core vector engine + adapters)
  • Backend client library for your chosen adapter (see tables above)
  • For LLM CLI: PyTorch 2.0+ and backend-specific packages

License

MIT

About

Compress Any LLM Up to 6x in One Command. Unified CLI for GGUF, GPTQ, and AWQ quantization.

Topics

quantization model-compression awq llm llama-cpp vllm gptq ollama gguf

Resources

Readme

License

MIT license
Activity

Stars

3 stars

Watchers

0 watching

Forks

1 fork
Report repository

Releases 1

TurboQuant v1.0.0 — 6x Compression for Vectors, Embeddings, and LLMs Latest
Mar 25, 2026

Packages

 
 
 

Contributors

Languages