diff --git a/.github/ignore-notebooks.txt b/.github/ignore-notebooks.txt index 5505268..61ba17d 100644 --- a/.github/ignore-notebooks.txt +++ b/.github/ignore-notebooks.txt @@ -7,4 +7,6 @@ 02_semantic_cache_optimization spring_ai_redis_rag.ipynb 00_litellm_proxy_redis.ipynb -04_redisvl_benchmarking_basics.ipynb \ No newline at end of file +04_redisvl_benchmarking_basics.ipynb +06_hnsw_to_svs_vamana_migration.ipynb +07_flat_to_svs_vamana_migration.ipynb \ No newline at end of file diff --git a/README.md b/README.md index a01de17..ca5c739 100644 --- a/README.md +++ b/README.md @@ -69,6 +69,8 @@ Need quickstarts to begin your Redis AI journey? | ๐Ÿ”ข **Data Type Support** - Shows how to convert a float32 index to float16 or integer dataypes | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/vector-search/03_dtype_support.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/vector-search/03_dtype_support.ipynb) | | ๐Ÿ“Š **Benchmarking Basics** - Overview of search benchmarking basics with RedisVL and Python multiprocessing | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/vector-search/04_redisvl_benchmarking_basics.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/vector-search/04_redisvl_benchmarking_basics.ipynb) | | ๐Ÿ“Š **Multi Vector Search** - Overview of multi vector queries with RedisVL | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/vector-search/05_multivector_search.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/vector-search/05_multivector_search.ipynb) | +| ๐Ÿ—œ๏ธ **HNSW to SVS-VAMANA Migration** - Showcase how to migrate HNSW indices to SVS-VAMANA | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/vector-search/06_hnsw_to_svs_vamana_migration.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/vector-search/06_hnsw_to_svs_vamana_migration.ipynb) | +| ๐Ÿ—œ๏ธ **FLAT to SVS-VAMANA Migration** - Showcase how to migrate FLAT indices to SVS-VAMANA | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/vector-search/07_flat_to_svs_vamana_migration.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/vector-search/07_flat_to_svs_vamana_migration.ipynb) | ### Retrieval Augmented Generation (RAG) diff --git a/python-recipes/vector-search/06_hnsw_to_svs_vamana_migration.ipynb b/python-recipes/vector-search/06_hnsw_to_svs_vamana_migration.ipynb new file mode 100644 index 0000000..8b2cee7 --- /dev/null +++ b/python-recipes/vector-search/06_hnsw_to_svs_vamana_migration.ipynb @@ -0,0 +1,1201 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", + "# Migrating from HNSW to SVS-VAMANA\n", + "\n", + "## Let's Begin!\n", + "\"Open\n", + "\n", + "This notebook demonstrates how to migrate existing HNSW vector indices to SVS-VAMANA for improved memory efficiency while maintaining search quality.\n", + "\n", + "## What You'll Learn\n", + "\n", + "- How to assess your current HNSW index for migration\n", + "- Step-by-step migration from HNSW to SVS-VAMANA\n", + "- Memory usage comparison and cost analysis\n", + "- Search quality validation between HNSW and SVS-VAMANA\n", + "- Performance benchmarking and recall comparison\n", + "- Migration decision framework for production systems\n", + "\n", + "## Prerequisites\n", + "\n", + "- Redis Stack 8.2.0+ with RediSearch 2.8.10+\n", + "- Existing HNSW index with substantial data (1000+ documents recommended)\n", + "- High-dimensional vectors (768+ dimensions for best compression benefits)\n", + "\n", + "## HNSW vs SVS-VAMANA\n", + "\n", + "**HNSW (Hierarchical Navigable Small World):**\n", + "- Excellent search quality and recall\n", + "- Fast query performance\n", + "- Higher memory usage (stores full-precision vectors)\n", + "- Good for applications prioritizing search quality\n", + "\n", + "**SVS-VAMANA:**\n", + "- Competitive search quality with compression\n", + "- Significant memory savings (50-75% reduction)\n", + "- Built-in vector compression (LeanVec, quantization)\n", + "- Ideal for large-scale deployments with cost constraints" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ๐Ÿ“‹ HNSW to SVS-VAMANA Migration Checklist\n", + "\n", + "**PRE-MIGRATION:**\n", + "- โ˜ Backup existing HNSW index data\n", + "- โ˜ Test migration on staging environment\n", + "- โ˜ Validate search quality with real queries\n", + "- โ˜ Measure baseline HNSW performance metrics\n", + "- โ˜ Plan rollback strategy\n", + "- โ˜ Document current HNSW parameters (M, EF_construction, EF_runtime)\n", + "\n", + "**MIGRATION:**\n", + "- โ˜ Create SVS-VAMANA index with tested configuration\n", + "- โ˜ Migrate data in batches during low-traffic periods\n", + "- โ˜ Monitor memory usage and indexing progress\n", + "- โ˜ Validate data integrity after migration\n", + "- โ˜ Test search functionality thoroughly\n", + "- โ˜ Compare recall metrics with baseline\n", + "\n", + "**POST-MIGRATION:**\n", + "- โ˜ Monitor search performance and quality\n", + "- โ˜ Track memory usage and cost savings\n", + "- โ˜ Update application configuration\n", + "- โ˜ Document new SVS-VAMANA settings\n", + "- โ˜ Clean up old HNSW index after validation period\n", + "- โ˜ Update monitoring and alerting thresholds\n", + "\n", + "**๐Ÿ’ก HNSW-SPECIFIC TIPS:**\n", + "- HNSW indices are more complex to rebuild than FLAT\n", + "- Consider the impact on applications using EF_runtime tuning\n", + "- SVS-VAMANA may have different optimal query parameters\n", + "- Test with your specific HNSW configuration (M, EF values)\n", + "- Monitor for 48-72 hours before removing HNSW index\n", + "- Keep compression settings documented for future reference" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ๐Ÿ“ฆ Installation & Setup\n", + "\n", + "This notebook uses **RedisVL vectorizers** for generating embeddings and **Redis Stack** for vector search.\n", + "\n", + "**Requirements:**\n", + "- Redis Stack 8.2.0+ with RediSearch 2.8.10+ (for SVS-VAMANA support)\n", + "- redisvl>=0.11.0 (required for SVS-VAMANA migration features and vectorizers)\n", + "- redis-py>=6.4.0 (required for compatibility with RedisVL 0.11.0+)\n", + "- numpy (for vector operations)\n", + "\n", + "**โš ๏ธ Important:** If you encounter Redis connection errors, upgrade redis-py: `pip install -U \"redis>=6.4.0\"`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Install Packages" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "/Users/nitin.kanukolanu/workspace/redis-vl-python/.venv/bin/python3: No module named pip\n", + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install git+https://github.com/redis/redis-vl-python.git \"redis>=6.4.0\" \"numpy>=1.21.0\" \"sentence-transformers>=2.2.0\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Install Redis Stack\n", + "\n", + "Later in this tutorial, Redis will be used to store, index, and query vector\n", + "embeddings and full text fields. **We need to have a Redis\n", + "instance available.**\n", + "\n", + "#### Local Redis\n", + "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [], + "source": [ + "# NBVAL_SKIP\n", + "%%sh\n", + "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n", + "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n", + "sudo apt-get update > /dev/null 2>&1\n", + "sudo apt-get install redis-stack-server > /dev/null 2>&1\n", + "redis-stack-server --daemonize yes" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Alternative Redis Access (Cloud, Docker, other)\n", + "There are many ways to get the necessary redis-stack instance running\n", + "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n", + "own version of Redis Enterprise running, that works too!\n", + "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n", + "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Define the Redis Connection URL\n", + "\n", + "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ“š Libraries imported successfully!\n" + ] + } + ], + "source": [ + "import os\n", + "import json\n", + "import numpy as np\n", + "import time\n", + "from typing import List, Dict, Any\n", + "\n", + "# Redis and RedisVL imports\n", + "import redis\n", + "from redisvl.index import SearchIndex\n", + "from redisvl.query import VectorQuery\n", + "from redisvl.redis.utils import array_to_buffer, buffer_to_array\n", + "from redisvl.utils import CompressionAdvisor\n", + "from redisvl.redis.connection import supports_svs\n", + "\n", + "# RedisVL Vectorizer imports\n", + "from redisvl.utils.vectorize import HFTextVectorizer\n", + "\n", + "# Replace values below with your own if using Redis Cloud instance\n", + "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", + "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", + "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", + "\n", + "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", + "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"\n", + "\n", + "print(\"๐Ÿ“š Libraries imported successfully!\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 1: Verify Redis and SVS Support\n", + "\n", + "First, let's ensure Redis Stack is running and supports SVS-VAMANA." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "โœ… Redis connection successful\n", + "๐Ÿ“Š Redis version: 8.2.2\n", + "โœ… SVS-VAMANA supported\n" + ] + } + ], + "source": [ + "# Test Redis connection and SVS support\n", + "try:\n", + " client = redis.Redis.from_url(REDIS_URL)\n", + " client.ping()\n", + " print(\"โœ… Redis connection successful\")\n", + " \n", + " # Check Redis version\n", + " redis_info = client.info()\n", + " redis_version = redis_info['redis_version']\n", + " print(f\"๐Ÿ“Š Redis version: {redis_version}\")\n", + " \n", + " # Check SVS support\n", + " if supports_svs(client):\n", + " print(\"โœ… SVS-VAMANA supported\")\n", + " else:\n", + " print(\"โŒ SVS-VAMANA not supported\")\n", + " print(\"Please ensure you're using Redis Stack 8.2.0+ with RediSearch 2.8.10+\")\n", + " \n", + "except Exception as e:\n", + " print(f\"โŒ Redis connection failed: {e}\")\n", + " print(\"Please ensure Redis Stack is running on localhost:6379\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 2: Load Sample Data\n", + "\n", + "We'll use the movie dataset to demonstrate the migration process." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ“ฝ๏ธ Loaded 20 movie records\n", + "Sample movie: Explosive Pursuit\n", + "Genres available: {'comedy', 'action'}\n", + "\n", + "๐Ÿ”ง Configuration:\n", + "Vector dimensions: 768\n", + "Dataset size: 20 movie documents\n", + "Vectorizer: RedisVL HFTextVectorizer\n" + ] + } + ], + "source": [ + "# Load the movies dataset\n", + "with open('resources/movies.json', 'r') as f:\n", + " movies_data = json.load(f)\n", + "\n", + "print(\n", + " f\"๐Ÿ“ฝ๏ธ Loaded {len(movies_data)} movie records\",\n", + " f\"Sample movie: {movies_data[0]['title']}\",\n", + " f\"Genres available: {set(movie['genre'] for movie in movies_data)}\",\n", + " sep=\"\\n\"\n", + ")\n", + "\n", + "# Configuration for demonstration \n", + "dims = 768 # Using all-mpnet-base-v2 model (768 dimensions)\n", + "num_docs = len(movies_data) # Use actual dataset size\n", + "\n", + "print(\n", + " f\"\\n๐Ÿ”ง Configuration:\",\n", + " f\"Vector dimensions: {dims}\",\n", + " f\"Dataset size: {num_docs} movie documents\",\n", + " f\"Vectorizer: RedisVL HFTextVectorizer\",\n", + " sep=\"\\n\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 3: Create HNSW Index\n", + "\n", + "First, we'll create an HNSW index with typical production settings." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Creating HNSW index with optimized settings...\n", + "โœ… Created HNSW index: hnsw_demo_index\n", + "\n", + "๐Ÿ”ง HNSW Configuration:\n", + "M (connections per node): 16\n", + "EF Construction: 200\n", + "EF Runtime: 10\n", + "Distance metric: cosine\n", + "Data type: float32\n" + ] + } + ], + "source": [ + "# Create HNSW schema with production-like settings\n", + "hnsw_schema = {\n", + " \"index\": {\n", + " \"name\": \"hnsw_demo_index\",\n", + " \"prefix\": \"demo:hnsw:\",\n", + " },\n", + " \"fields\": [\n", + " {\"name\": \"movie_id\", \"type\": \"tag\"},\n", + " {\"name\": \"title\", \"type\": \"text\"},\n", + " {\"name\": \"genre\", \"type\": \"tag\"},\n", + " {\"name\": \"rating\", \"type\": \"numeric\"},\n", + " {\"name\": \"description\", \"type\": \"text\"},\n", + " {\n", + " \"name\": \"embedding\",\n", + " \"type\": \"vector\",\n", + " \"attrs\": {\n", + " \"dims\": dims,\n", + " \"algorithm\": \"hnsw\",\n", + " \"datatype\": \"float32\",\n", + " \"distance_metric\": \"cosine\",\n", + " \"m\": 16, # Number of bi-directional links for each node\n", + " \"ef_construction\": 200, # Size of dynamic candidate list\n", + " \"ef_runtime\": 10 # Size of dynamic candidate list during search\n", + " }\n", + " }\n", + " ]\n", + "}\n", + "\n", + "print(\"Creating HNSW index with optimized settings...\")\n", + "hnsw_index = SearchIndex.from_dict(hnsw_schema, redis_url=REDIS_URL)\n", + "hnsw_index.create(overwrite=True)\n", + "print(f\"โœ… Created HNSW index: {hnsw_index.name}\")\n", + "\n", + "# Display HNSW configuration\n", + "print(\n", + " \"\\n๐Ÿ”ง HNSW Configuration:\",\n", + " f\"M (connections per node): 16\",\n", + " f\"EF Construction: 200\",\n", + " f\"EF Runtime: 10\",\n", + " f\"Distance metric: cosine\",\n", + " f\"Data type: float32\",\n", + " sep=\"\\n\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 4: Generate Embeddings and Load HNSW Index\n", + "\n", + "Generate embeddings for movie descriptions and populate the HNSW index." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ”„ Generating embeddings for movie descriptions...\n", + "๐Ÿš€ Using RedisVL HFTextVectorizer...\n", + "12:13:05 sentence_transformers.SentenceTransformer INFO Use pytorch device_name: mps\n", + "12:13:05 sentence_transformers.SentenceTransformer INFO Load pretrained SentenceTransformer: sentence-transformers/all-mpnet-base-v2\n", + "โœ… Generated 20 real embeddings using RedisVL HFTextVectorizer\n", + "๐Ÿ“Š Embedding shape: (20, 768)\n" + ] + } + ], + "source": [ + "# Generate embeddings using RedisVL vectorizers\n", + "print(\"๐Ÿ”„ Generating embeddings for movie descriptions...\")\n", + "\n", + "descriptions = [movie['description'] for movie in movies_data]\n", + "\n", + "# Use RedisVL HFTextVectorizer\n", + "print(\"๐Ÿš€ Using RedisVL HFTextVectorizer...\")\n", + "vectorizer = HFTextVectorizer(\n", + " model=\"sentence-transformers/all-mpnet-base-v2\", # 768 dimensions\n", + ")\n", + "\n", + "# Generate embeddings using RedisVL vectorizer\n", + "embeddings = vectorizer.embed_many(descriptions)\n", + "embeddings = np.array(embeddings, dtype=np.float32)\n", + "\n", + "print(f\"โœ… Generated {len(embeddings)} real embeddings using RedisVL HFTextVectorizer\")\n", + "print(f\"๐Ÿ“Š Embedding shape: {embeddings.shape}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ“ฆ Prepared 20 documents for indexing\n" + ] + } + ], + "source": [ + "# Prepare data for loading into HNSW index\n", + "sample_data = []\n", + "for i, movie in enumerate(movies_data):\n", + " sample_data.append({\n", + " 'movie_id': str(movie['id']),\n", + " 'title': movie['title'],\n", + " 'genre': movie['genre'],\n", + " 'rating': movie['rating'],\n", + " 'description': movie['description'],\n", + " 'embedding': array_to_buffer(embeddings[i].astype(np.float32), dtype='float32')\n", + " })\n", + "\n", + "print(f\"๐Ÿ“ฆ Prepared {len(sample_data)} documents for indexing\")" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ“ฅ Loading data into HNSW index...\n", + " Loaded 20/20 documents\n", + "โณ Waiting for HNSW indexing to complete...\n", + "\n", + "โœ… HNSW index loaded with 20 documents\n", + "Index size: 3.2259750366210938 MB\n", + "Indexing time: ~5 seconds (HNSW graph construction)\n" + ] + } + ], + "source": [ + "# Load data into HNSW index\n", + "print(\"๐Ÿ“ฅ Loading data into HNSW index...\")\n", + "batch_size = 100 # Process in batches\n", + "\n", + "for i in range(0, len(sample_data), batch_size):\n", + " batch = sample_data[i:i+batch_size]\n", + " hnsw_index.load(batch)\n", + " print(f\" Loaded {min(i+batch_size, len(sample_data))}/{len(sample_data)} documents\")\n", + "\n", + "# Wait for indexing to complete\n", + "print(\"โณ Waiting for HNSW indexing to complete...\")\n", + "time.sleep(5) # HNSW indexing takes longer than FLAT\n", + "\n", + "hnsw_info = hnsw_index.info()\n", + "print(\n", + " f\"\\nโœ… HNSW index loaded with {hnsw_info['num_docs']} documents\",\n", + " f\"Index size: {hnsw_info.get('vector_index_sz_mb', 'N/A')} MB\",\n", + " f\"Indexing time: ~5 seconds (HNSW graph construction)\",\n", + " sep=\"\\n\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 5: Get Compression Recommendation\n", + "\n", + "Use the CompressionAdvisor to get optimal SVS-VAMANA settings for our data." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ” Analyzing data for optimal compression settings...\n", + "\n", + "๐Ÿ“Š Compression Recommendations:\n", + "\n", + "๐Ÿ—œ๏ธ Memory Priority:\n", + " Algorithm: svs-vamana\n", + " Compression: LVQ4\n", + " Datatype: float32\n", + " Dimensions: 768 โ†’ None\n", + "\n", + "โš–๏ธ Balanced Priority:\n", + " Algorithm: svs-vamana\n", + " Compression: LVQ4x4\n", + " Datatype: float32\n", + " Dimensions: 768 โ†’ None\n", + "\n", + "โšก Performance Priority:\n", + " Algorithm: svs-vamana\n", + " Compression: LVQ4x4\n", + " Datatype: float32\n", + " Dimensions: 768 โ†’ None\n", + "\n", + "โœ… Selected configuration: Memory Priority\n", + "Expected memory reduction: ~0.0% from dimension reduction\n", + "Additional savings from float32 compression\n" + ] + } + ], + "source": [ + "# Get compression recommendation\n", + "print(\"๐Ÿ” Analyzing data for optimal compression settings...\")\n", + "\n", + "# Get recommendations for different priorities\n", + "memory_config = CompressionAdvisor.recommend(dims=dims, priority=\"memory\")\n", + "balanced_config = CompressionAdvisor.recommend(dims=dims, priority=\"balanced\")\n", + "performance_config = CompressionAdvisor.recommend(dims=dims, priority=\"performance\")\n", + "\n", + "print(\n", + " \"\\n๐Ÿ“Š Compression Recommendations:\",\n", + " \"\",\n", + " \"๐Ÿ—œ๏ธ Memory Priority:\",\n", + " f\" Algorithm: {memory_config.algorithm}\",\n", + " f\" Compression: {memory_config.compression if hasattr(memory_config, 'compression') else 'None'}\",\n", + " f\" Datatype: {memory_config.datatype}\",\n", + " f\" Dimensions: {dims} โ†’ {memory_config.reduce if hasattr(memory_config, 'reduce') else dims}\",\n", + " \"\",\n", + " \"โš–๏ธ Balanced Priority:\",\n", + " f\" Algorithm: {balanced_config.algorithm}\",\n", + " f\" Compression: {balanced_config.compression if hasattr(balanced_config, 'compression') else 'None'}\",\n", + " f\" Datatype: {balanced_config.datatype}\",\n", + " f\" Dimensions: {dims} โ†’ {balanced_config.reduce if hasattr(balanced_config, 'reduce') else dims}\",\n", + " \"\",\n", + " \"โšก Performance Priority:\",\n", + " f\" Algorithm: {performance_config.algorithm}\",\n", + " f\" Compression: {performance_config.compression if hasattr(performance_config, 'compression') else 'None'}\",\n", + " f\" Datatype: {performance_config.datatype}\",\n", + " f\" Dimensions: {dims} โ†’ {performance_config.reduce if hasattr(performance_config, 'reduce') else dims}\",\n", + " sep=\"\\n\"\n", + ")\n", + "\n", + "# Select configuration (using memory priority for maximum savings)\n", + "selected_config = memory_config\n", + "# Use reduce if it exists and is not None, otherwise use original dims\n", + "target_dims = selected_config.reduce if (hasattr(selected_config, 'reduce') and selected_config.reduce is not None) else dims\n", + "target_dtype = selected_config.datatype\n", + "\n", + "print(\n", + " f\"\\nโœ… Selected configuration: Memory Priority\",\n", + " f\"Expected memory reduction: ~{((dims - target_dims) / dims * 100):.1f}% from dimension reduction\",\n", + " f\"Additional savings from {selected_config.datatype} compression\",\n", + " sep=\"\\n\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 6: Create SVS-VAMANA Index\n", + "\n", + "Create the SVS-VAMANA index with the recommended compression settings." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Creating SVS-VAMANA index with compression...\n", + "โœ… Created SVS-VAMANA index: svs_demo_index\n", + "Compression: LVQ4\n", + "Datatype: float32\n", + "Dimensions: 768 โ†’ 768\n" + ] + } + ], + "source": [ + "# Create SVS-VAMANA schema with compression\n", + "svs_schema = {\n", + " \"index\": {\n", + " \"name\": \"svs_demo_index\",\n", + " \"prefix\": \"demo:svs:\",\n", + " },\n", + " \"fields\": [\n", + " {\"name\": \"movie_id\", \"type\": \"tag\"},\n", + " {\"name\": \"title\", \"type\": \"text\"},\n", + " {\"name\": \"genre\", \"type\": \"tag\"},\n", + " {\"name\": \"rating\", \"type\": \"numeric\"},\n", + " {\"name\": \"description\", \"type\": \"text\"},\n", + " {\n", + " \"name\": \"embedding\",\n", + " \"type\": \"vector\",\n", + " \"attrs\": {\n", + " \"dims\": target_dims, # Use reduced dimensions\n", + " \"algorithm\": \"svs-vamana\",\n", + " \"datatype\": selected_config.datatype,\n", + " \"distance_metric\": \"cosine\"\n", + " # Note: Don't include the full selected_config to avoid dims/reduce conflict\n", + " }\n", + " }\n", + " ]\n", + "}\n", + "\n", + "print(\"Creating SVS-VAMANA index with compression...\")\n", + "svs_index = SearchIndex.from_dict(svs_schema, redis_url=REDIS_URL)\n", + "svs_index.create(overwrite=True)\n", + "print(\n", + " f\"โœ… Created SVS-VAMANA index: {svs_index.name}\",\n", + " f\"Compression: {selected_config.compression if hasattr(selected_config, 'compression') else 'None'}\",\n", + " f\"Datatype: {selected_config.datatype}\",\n", + " f\"Dimensions: {dims} โ†’ {target_dims}\",\n", + " sep=\"\\n\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 7: Migrate Data from HNSW to SVS-VAMANA\n", + "\n", + "Extract data from the HNSW index and migrate it to SVS-VAMANA with compression." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ”„ Extracting data from HNSW index...\n", + "Found 20 documents to migrate\n", + "Prepared 20 documents for migration\n" + ] + } + ], + "source": [ + "# Extract data from HNSW index\n", + "print(\"๐Ÿ”„ Extracting data from HNSW index...\")\n", + "\n", + "client = redis.Redis.from_url(REDIS_URL)\n", + "keys = client.keys(\"demo:hnsw:*\")\n", + "print(f\"Found {len(keys)} documents to migrate\")\n", + "\n", + "# Process and transform data for SVS index\n", + "svs_data = []\n", + "\n", + "for key in keys:\n", + " doc_data = client.hgetall(key)\n", + " \n", + " if b'embedding' in doc_data:\n", + " # Extract original vector from HNSW index\n", + " original_vector = np.array(buffer_to_array(doc_data[b'embedding'], dtype='float32'))\n", + " \n", + " # Apply dimensionality reduction if needed (LeanVec)\n", + " if target_dims < dims:\n", + " vector = original_vector[:target_dims]\n", + " else:\n", + " vector = original_vector\n", + " \n", + " # Convert to target datatype\n", + " if target_dtype == 'float16':\n", + " vector = vector.astype(np.float16)\n", + " \n", + " svs_data.append({\n", + " \"movie_id\": doc_data[b'movie_id'].decode(),\n", + " \"title\": doc_data[b'title'].decode(),\n", + " \"genre\": doc_data[b'genre'].decode(),\n", + " \"rating\": int(doc_data[b'rating'].decode()),\n", + " \"description\": doc_data[b'description'].decode(),\n", + " \"embedding\": array_to_buffer(vector, dtype=target_dtype)\n", + " })\n", + "\n", + "print(f\"Prepared {len(svs_data)} documents for migration\")" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ“ฅ Loading data into SVS-VAMANA index...\n", + " Migrated 20/20 documents\n", + "โณ Waiting for SVS-VAMANA indexing to complete...\n", + "\n", + "โœ… Migration complete! SVS index has 20 documents\n", + "Index size: 3.017791748046875 MB\n" + ] + } + ], + "source": [ + "# Load data into SVS index\n", + "print(\"๐Ÿ“ฅ Loading data into SVS-VAMANA index...\")\n", + "batch_size = 100 # Define batch size for migration\n", + "\n", + "if len(svs_data) > 0:\n", + " for i in range(0, len(svs_data), batch_size):\n", + " batch = svs_data[i:i+batch_size]\n", + " svs_index.load(batch)\n", + " print(f\" Migrated {min(i+batch_size, len(svs_data))}/{len(svs_data)} documents\")\n", + "\n", + " # Wait for indexing to complete\n", + " print(\"โณ Waiting for SVS-VAMANA indexing to complete...\")\n", + " time.sleep(5)\n", + "\n", + " svs_info = svs_index.info()\n", + " print(\n", + " f\"\\nโœ… Migration complete! SVS index has {svs_info['num_docs']} documents\",\n", + " f\"Index size: {svs_info.get('vector_index_sz_mb', 'N/A')} MB\",\n", + " sep=\"\\n\"\n", + " )\n", + "else:\n", + " print(\"โš ๏ธ No data to migrate. Make sure the HNSW index was populated first.\")\n", + " print(\" Run the previous cells to load data into the HNSW index.\")\n", + " svs_info = svs_index.info()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 8: Compare Memory Usage\n", + "\n", + "Analyze the memory savings achieved through the HNSW to SVS-VAMANA migration." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ“Š Memory Usage Comparison\n", + "========================================\n", + "Original HNSW index: 3.23 MB\n", + "SVS-VAMANA index: 3.02 MB\n", + "\n", + "๐Ÿ’ฐ Memory savings: 6.5%\n", + "Absolute reduction: 0.21 MB\n" + ] + } + ], + "source": [ + "# Helper function to extract memory info\n", + "def get_memory_mb(index_info):\n", + " \"\"\"Extract memory usage in MB from index info\"\"\"\n", + " memory = index_info.get('vector_index_sz_mb', 0)\n", + " if isinstance(memory, str):\n", + " try:\n", + " return float(memory)\n", + " except ValueError:\n", + " return 0.0\n", + " return float(memory)\n", + "\n", + "# Get memory usage\n", + "hnsw_memory = get_memory_mb(hnsw_info)\n", + "svs_memory = get_memory_mb(svs_info)\n", + "\n", + "print(\n", + " \"๐Ÿ“Š Memory Usage Comparison\",\n", + " \"=\" * 40,\n", + " f\"Original HNSW index: {hnsw_memory:.2f} MB\",\n", + " f\"SVS-VAMANA index: {svs_memory:.2f} MB\",\n", + " \"\",\n", + " sep=\"\\n\"\n", + ")\n", + "\n", + "if hnsw_memory > 0:\n", + " if svs_memory > 0:\n", + " savings = ((hnsw_memory - svs_memory) / hnsw_memory) * 100\n", + " print(\n", + " f\"๐Ÿ’ฐ Memory savings: {savings:.1f}%\",\n", + " f\"Absolute reduction: {hnsw_memory - svs_memory:.2f} MB\",\n", + " sep=\"\\n\"\n", + " )\n", + " else:\n", + " print(\"โณ SVS index still indexing - memory comparison pending\")\n", + "else:\n", + " print(\"โš ๏ธ Memory information not available\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 9: Validate Search Quality\n", + "\n", + "Compare search quality between HNSW and SVS-VAMANA to ensure the migration maintains acceptable recall." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ” Generating test queries for quality validation...\n", + "Generated 10 test queries\n" + ] + } + ], + "source": [ + "# Generate test queries\n", + "print(\"๐Ÿ” Generating test queries for quality validation...\")\n", + "\n", + "np.random.seed(123) # For reproducible test queries\n", + "num_test_queries = 10\n", + "test_queries = []\n", + "\n", + "for i in range(num_test_queries):\n", + " # Create test query vectors\n", + " query_vec = np.random.random(dims).astype(np.float32)\n", + " query_vec = query_vec / np.linalg.norm(query_vec) # Normalize\n", + " test_queries.append(query_vec)\n", + "\n", + "print(f\"Generated {len(test_queries)} test queries\")" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ” Testing HNSW search quality...\n", + "HNSW search completed in 0.010 seconds\n" + ] + } + ], + "source": [ + "# Test HNSW search quality\n", + "print(\"๐Ÿ” Testing HNSW search quality...\")\n", + "\n", + "hnsw_results = []\n", + "hnsw_start = time.time()\n", + "\n", + "for query_vec in test_queries:\n", + " query = VectorQuery(\n", + " vector=query_vec,\n", + " vector_field_name=\"embedding\",\n", + " return_fields=[\"movie_id\", \"title\", \"genre\"],\n", + " dtype=\"float32\",\n", + " num_results=10\n", + " )\n", + " results = hnsw_index.query(query)\n", + " hnsw_results.append([doc[\"movie_id\"] for doc in results])\n", + "\n", + "hnsw_time = time.time() - hnsw_start\n", + "print(f\"HNSW search completed in {hnsw_time:.3f} seconds\")" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ” Testing SVS-VAMANA search quality...\n", + "SVS-VAMANA search completed in 0.009 seconds\n" + ] + } + ], + "source": [ + "# Test SVS-VAMANA search quality\n", + "print(\"๐Ÿ” Testing SVS-VAMANA search quality...\")\n", + "\n", + "svs_results = []\n", + "svs_start = time.time()\n", + "\n", + "for i, query_vec in enumerate(test_queries):\n", + " # Adjust query vector for SVS index (handle dimensionality reduction)\n", + " if target_dims < dims:\n", + " svs_query_vec = query_vec[:target_dims]\n", + " else:\n", + " svs_query_vec = query_vec\n", + " \n", + " if target_dtype == 'float16':\n", + " svs_query_vec = svs_query_vec.astype(np.float16)\n", + " \n", + " query = VectorQuery(\n", + " vector=svs_query_vec,\n", + " vector_field_name=\"embedding\",\n", + " return_fields=[\"movie_id\", \"title\", \"genre\"],\n", + " dtype=target_dtype,\n", + " num_results=10\n", + " )\n", + " results = svs_index.query(query)\n", + " svs_results.append([doc[\"movie_id\"] for doc in results])\n", + "\n", + "svs_time = time.time() - svs_start\n", + "print(f\"SVS-VAMANA search completed in {svs_time:.3f} seconds\")" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ“Š Search Quality Comparison\n", + "========================================\n", + "HNSW (baseline): 100% recall (exact graph-based search)\n", + "SVS-VAMANA Recall@5: 100.0% (vs HNSW baseline)\n", + "SVS-VAMANA Recall@10: 100.0% (vs HNSW baseline)\n", + "\n", + "โฑ๏ธ Performance Comparison:\n", + "HNSW query time: 0.010s (1.0ms per query)\n", + "SVS-VAMANA query time: 0.009s (0.9ms per query)\n", + "Speed difference: +5.9%\n", + "\n", + "๐ŸŽฏ Quality Assessment: ๐ŸŸข Excellent - Minimal quality loss\n" + ] + } + ], + "source": [ + "# Calculate recall and performance metrics\n", + "def calculate_recall(reference_results, test_results, k=10):\n", + " \"\"\"Calculate recall@k between two result sets\"\"\"\n", + " if not reference_results or not test_results:\n", + " return 0.0\n", + " \n", + " total_recall = 0.0\n", + " for ref, test in zip(reference_results, test_results):\n", + " ref_set = set(ref[:k])\n", + " test_set = set(test[:k])\n", + " if len(ref_set) > 0:\n", + " recall = len(ref_set.intersection(test_set)) / len(ref_set)\n", + " total_recall += recall\n", + " \n", + " return total_recall / len(reference_results)\n", + "\n", + "# Calculate metrics\n", + "recall_at_5 = calculate_recall(hnsw_results, svs_results, k=5)\n", + "recall_at_10 = calculate_recall(hnsw_results, svs_results, k=10)\n", + "\n", + "print(\n", + " \"๐Ÿ“Š Search Quality Comparison\",\n", + " \"=\" * 40,\n", + " \"HNSW (baseline): 100% recall (exact graph-based search)\",\n", + " f\"SVS-VAMANA Recall@5: {recall_at_5*100:.1f}% (vs HNSW baseline)\",\n", + " f\"SVS-VAMANA Recall@10: {recall_at_10*100:.1f}% (vs HNSW baseline)\",\n", + " \"\",\n", + " \"โฑ๏ธ Performance Comparison:\",\n", + " f\"HNSW query time: {hnsw_time:.3f}s ({hnsw_time/num_test_queries*1000:.1f}ms per query)\",\n", + " f\"SVS-VAMANA query time: {svs_time:.3f}s ({svs_time/num_test_queries*1000:.1f}ms per query)\",\n", + " f\"Speed difference: {((hnsw_time - svs_time) / hnsw_time * 100):+.1f}%\",\n", + " sep=\"\\n\"\n", + ")\n", + "\n", + "# Quality assessment\n", + "if recall_at_10 >= 0.95:\n", + " quality_assessment = \"๐ŸŸข Excellent - Minimal quality loss\"\n", + "elif recall_at_10 >= 0.90:\n", + " quality_assessment = \"๐ŸŸก Good - Acceptable quality for most applications\"\n", + "elif recall_at_10 >= 0.80:\n", + " quality_assessment = \"๐ŸŸ  Fair - Consider if quality requirements are flexible\"\n", + "else:\n", + " quality_assessment = \"๐Ÿ”ด Poor - Migration not recommended\"\n", + "\n", + "print(f\"\\n๐ŸŽฏ Quality Assessment: {quality_assessment}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 10: Migration Decision Framework\n", + "\n", + "Based on the analysis, determine if migration is recommended." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿค” Migration Decision Analysis\n", + "========================================\n", + "\n", + "๐Ÿ“Š Criteria Evaluation:\n", + "Memory savings: 6.5% โœ… (threshold: 5%)\n", + "Search quality: 1.000 โœ… (threshold: 0.85)\n", + "\n", + "๐ŸŽฏ Migration Recommendation: ๐ŸŸข RECOMMENDED\n", + "๐Ÿ’ญ Reasoning: Migration provides significant memory savings while maintaining good search quality.\n" + ] + } + ], + "source": [ + "# Migration decision logic\n", + "memory_savings_threshold = 5 # Minimum % memory savings\n", + "recall_threshold = 0.85 # Minimum 85% recall@10\n", + "\n", + "memory_savings_pct = ((hnsw_memory - svs_memory) / hnsw_memory * 100) if hnsw_memory > 0 and svs_memory > 0 else 0\n", + "meets_memory_threshold = memory_savings_pct >= memory_savings_threshold\n", + "meets_quality_threshold = recall_at_10 >= recall_threshold\n", + "\n", + "print(\n", + " \"๐Ÿค” Migration Decision Analysis\",\n", + " \"=\" * 40,\n", + " \"\",\n", + " \"๐Ÿ“Š Criteria Evaluation:\",\n", + " f\"Memory savings: {memory_savings_pct:.1f}% {'โœ…' if meets_memory_threshold else 'โŒ'} (threshold: {memory_savings_threshold}%)\",\n", + " f\"Search quality: {recall_at_10:.3f} {'โœ…' if meets_quality_threshold else 'โŒ'} (threshold: {recall_threshold})\",\n", + " \"\",\n", + " sep=\"\\n\"\n", + ")\n", + "\n", + "if meets_memory_threshold and meets_quality_threshold:\n", + " recommendation = \"๐ŸŸข RECOMMENDED\"\n", + " reasoning = \"Migration provides significant memory savings while maintaining good search quality.\"\n", + "elif meets_memory_threshold and not meets_quality_threshold:\n", + " recommendation = \"๐ŸŸก CONDITIONAL\"\n", + " reasoning = \"Good memory savings but reduced search quality. Consider if your application can tolerate lower recall.\"\n", + "elif not meets_memory_threshold and meets_quality_threshold:\n", + " recommendation = \"๐ŸŸ  LIMITED BENEFIT\"\n", + " reasoning = \"Search quality is maintained but memory savings are minimal. Migration may not be worth the effort.\"\n", + "else:\n", + " recommendation = \"๐Ÿ”ด NOT RECOMMENDED\"\n", + " reasoning = \"Insufficient memory savings and/or poor search quality. Consider alternative optimization strategies.\"\n", + "\n", + "print(\n", + " f\"๐ŸŽฏ Migration Recommendation: {recommendation}\",\n", + " f\"๐Ÿ’ญ Reasoning: {reasoning}\",\n", + " sep=\"\\n\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 12: Cleanup\n", + "\n", + "Clean up the demonstration indices." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿงน Cleaning up demonstration indices...\n", + "โœ… Deleted HNSW demonstration index\n", + "โœ… Deleted SVS-VAMANA demonstration index\n", + "\n", + "๐ŸŽ‰ HNSW to SVS-VAMANA migration demonstration complete!\n", + "\n", + "Next steps:\n", + "1. Apply learnings to your production HNSW indices\n", + "2. Test with your actual query patterns and data\n", + "3. Monitor performance in your environment\n", + "4. Consider gradual rollout strategy\n", + "5. Evaluate impact on applications using HNSW-specific features\n" + ] + } + ], + "source": [ + "print(\"๐Ÿงน Cleaning up demonstration indices...\")\n", + "\n", + "# Clean up HNSW index\n", + "try:\n", + " hnsw_index.delete(drop=True)\n", + " print(\"โœ… Deleted HNSW demonstration index\")\n", + "except Exception as e:\n", + " print(f\"โš ๏ธ Failed to delete HNSW index: {e}\")\n", + "\n", + "# Clean up SVS index\n", + "try:\n", + " svs_index.delete(drop=True)\n", + " print(\"โœ… Deleted SVS-VAMANA demonstration index\")\n", + "except Exception as e:\n", + " print(f\"โš ๏ธ Failed to delete SVS index: {e}\")\n", + "\n", + "print(\n", + " \"\\n๐ŸŽ‰ HNSW to SVS-VAMANA migration demonstration complete!\",\n", + " \"\\nNext steps:\",\n", + " \"1. Apply learnings to your production HNSW indices\",\n", + " \"2. Test with your actual query patterns and data\",\n", + " \"3. Monitor performance in your environment\",\n", + " \"4. Consider gradual rollout strategy\",\n", + " \"5. Evaluate impact on applications using HNSW-specific features\",\n", + " sep=\"\\n\"\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.6" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/python-recipes/vector-search/07_flat_to_svs_vamana_migration.ipynb b/python-recipes/vector-search/07_flat_to_svs_vamana_migration.ipynb new file mode 100644 index 0000000..6b7789f --- /dev/null +++ b/python-recipes/vector-search/07_flat_to_svs_vamana_migration.ipynb @@ -0,0 +1,1090 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", + "# Migrating from FLAT to SVS-VAMANA\n", + "\n", + "## Let's Begin!\n", + "\"Open\n", + "\n", + "This notebook demonstrates how to migrate existing FLAT vector indices to SVS-VAMANA for improved memory efficiency and cost savings.\n", + "\n", + "## What You'll Learn\n", + "\n", + "- How to assess your current FLAT index for migration\n", + "- Step-by-step migration from FLAT to SVS-VAMANA\n", + "- Memory usage comparison and cost analysis\n", + "- Search quality validation\n", + "- Performance benchmarking\n", + "- Migration decision framework\n", + "\n", + "## Prerequisites\n", + "\n", + "- Redis Stack 8.2.0+ with RediSearch 2.8.10+\n", + "- Existing vector index with substantial data (1000+ documents recommended)\n", + "- Vector embeddings (768 dimensions using sentence-transformers/all-mpnet-base-v2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ๐Ÿ“‹ FLAT to SVS-VAMANA Migration Checklist\n", + "\n", + "**PRE-MIGRATION:**\n", + "- โ˜ Backup existing FLAT index data\n", + "- โ˜ Test migration on staging environment\n", + "- โ˜ Validate search quality with real queries\n", + "- โ˜ Measure baseline FLAT performance metrics\n", + "- โ˜ Plan rollback strategy\n", + "- โ˜ Document current FLAT index configuration\n", + "\n", + "**MIGRATION:**\n", + "- โ˜ Create SVS-VAMANA index with tested configuration\n", + "- โ˜ Migrate data in batches during low-traffic periods\n", + "- โ˜ Monitor memory usage and indexing progress\n", + "- โ˜ Validate data integrity after migration\n", + "- โ˜ Test search functionality thoroughly\n", + "- โ˜ Compare recall metrics with baseline\n", + "\n", + "**POST-MIGRATION:**\n", + "- โ˜ Monitor search performance and quality\n", + "- โ˜ Track memory usage and cost savings\n", + "- โ˜ Update application configuration\n", + "- โ˜ Document new SVS-VAMANA settings\n", + "- โ˜ Clean up old FLAT index after validation period\n", + "- โ˜ Update monitoring and alerting thresholds\n", + "\n", + "**๐Ÿ’ก FLAT-SPECIFIC TIPS:**\n", + "- FLAT indices are simpler to migrate than HNSW (no graph structure)\n", + "- FLAT provides 100% recall, so focus on acceptable recall threshold for SVS-VAMANA\n", + "- SVS-VAMANA will be faster than FLAT for large datasets\n", + "- Memory savings are most significant with FLAT migrations\n", + "- Consider using compression for maximum memory reduction\n", + "- Test query performance improvements with your dataset size" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ๐Ÿ“ฆ Installation & Setup\n", + "\n", + "This notebook uses **RedisVL vectorizers** for generating embeddings and **Redis Stack** for vector search.\n", + "\n", + "**Requirements:**\n", + "- Redis Stack 8.2.0+ with RediSearch 2.8.10+ (for SVS-VAMANA support)\n", + "- redisvl>=0.11.0 (required for SVS-VAMANA migration features and vectorizers)\n", + "- redis-py>=6.4.0 (required for compatibility with RedisVL 0.11.0+)\n", + "- numpy (for vector operations)\n", + "\n", + "**โš ๏ธ Important:** If you encounter Redis connection errors, upgrade redis-py: `pip install -U \"redis>=6.4.0\"`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Install Packages" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%pip install git+https://github.com/redis/redis-vl-python.git \"redis>=6.4.0\" \"numpy>=1.21.0\" \"sentence-transformers>=2.2.0\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Install Redis Stack\n", + "\n", + "Later in this tutorial, Redis will be used to store, index, and query vector\n", + "embeddings and full text fields. **We need to have a Redis\n", + "instance available.**\n", + "\n", + "#### Local Redis\n", + "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# NBVAL_SKIP\n", + "%%sh\n", + "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n", + "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n", + "sudo apt-get update > /dev/null 2>&1\n", + "sudo apt-get install redis-stack-server > /dev/null 2>&1\n", + "redis-stack-server --daemonize yes" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Alternative Redis Access (Cloud, Docker, other)\n", + "There are many ways to get the necessary redis-stack instance running\n", + "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n", + "own version of Redis Enterprise running, that works too!\n", + "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n", + "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Define the Redis Connection URL\n", + "\n", + "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import sys\n", + "import subprocess\n", + "# Required imports from redis-vl\n", + "import numpy as np\n", + "import time\n", + "from redisvl.index import SearchIndex\n", + "from redisvl.query import VectorQuery\n", + "from redisvl.redis.utils import array_to_buffer, buffer_to_array\n", + "from redisvl.utils import CompressionAdvisor\n", + "from redisvl.redis.connection import supports_svs\n", + "import redis\n", + "\n", + "# RedisVL Vectorizer imports\n", + "from redisvl.utils.vectorize import HFTextVectorizer\n", + "\n", + "# Replace values below with your own if using Redis Cloud instance\n", + "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", + "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", + "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", + "\n", + "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", + "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 1: Verify SVS-VAMANA Support\n", + "\n", + "First, let's ensure your Redis environment supports SVS-VAMANA." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "โœ… Redis connection successful\n", + "โœ… SVS-VAMANA supported\n", + " Ready for migration!\n" + ] + } + ], + "source": [ + "# Check Redis connection and SVS support\n", + "REDIS_URL = \"redis://localhost:6379\"\n", + "\n", + "try:\n", + " client = redis.Redis.from_url(REDIS_URL)\n", + " client.ping()\n", + " print(\"โœ… Redis connection successful\")\n", + " \n", + " if supports_svs(client):\n", + " print(\"โœ… SVS-VAMANA supported\")\n", + " print(\" Ready for migration!\")\n", + " else:\n", + " print(\"โŒ SVS-VAMANA not supported\")\n", + " print(\" Requires Redis >= 8.2.0 with RediSearch >= 2.8.10\")\n", + " print(\" Please upgrade Redis Stack before proceeding\")\n", + " \n", + "except Exception as e:\n", + " print(f\"โŒ Redis connection failed: {e}\")\n", + " print(\" Please ensure Redis is running and accessible\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 2: Assess Your Current Index\n", + "\n", + "For this demonstration, we'll create a sample FLAT index. In practice, you would analyze your existing index." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ“ฅ Loading sample movie data...\n", + "Loaded 20 movie records\n", + "Sample movie: Explosive Pursuit - A daring cop chases a notorious criminal across the city in a high-stakes game of cat and mouse.\n" + ] + } + ], + "source": [ + "# Download sample data from redis-ai-resources\n", + "print(\"๐Ÿ“ฅ Loading sample movie data...\")\n", + "import os\n", + "import json\n", + "\n", + "# Load the movies dataset\n", + "url = \"resources/movies.json\"\n", + "with open(\"resources/movies.json\", \"r\") as f:\n", + " movies_data = json.load(f)\n", + "\n", + "print(f\"Loaded {len(movies_data)} movie records\")\n", + "print(f\"Sample movie: {movies_data[0]['title']} - {movies_data[0]['description']}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ“Š Migration Assessment\n", + "Vector dimensions: 768\n", + "Dataset size: 20 movie documents\n", + "Data includes: title, genre, rating, description\n", + "Vectorizer: RedisVL HFTextVectorizer\n" + ] + } + ], + "source": [ + "# Configuration for demonstration \n", + "dims = 768 # Using all-mpnet-base-v2 model (768 dimensions)\n", + "\n", + "num_docs = len(movies_data) # Use actual dataset size\n", + "\n", + "print(\n", + " \"๐Ÿ“Š Migration Assessment\",\n", + " f\"Vector dimensions: {dims}\",\n", + " f\"Dataset size: {num_docs} movie documents\",\n", + " \"Data includes: title, genre, rating, description\",\n", + " f\"Vectorizer: RedisVL HFTextVectorizer\",\n", + " sep=\"\\n\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "Next, let's configure a smaple FLAT index. Notice the algorithm value, dims value, and datatype value under fields." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Creating sample FLAT index...\n", + "12:00:03 redisvl.index.index INFO Index already exists, overwriting.\n", + "โœ… Created FLAT index: migration_demo_flat\n" + ] + } + ], + "source": [ + "flat_schema = {\n", + " \"index\": {\n", + " \"name\": \"migration_demo_flat\",\n", + " \"prefix\": \"demo:flat:\",\n", + " },\n", + " \"fields\": [\n", + " {\"name\": \"movie_id\", \"type\": \"tag\"},\n", + " {\"name\": \"title\", \"type\": \"text\"},\n", + " {\"name\": \"genre\", \"type\": \"tag\"},\n", + " {\"name\": \"rating\", \"type\": \"numeric\"},\n", + " {\"name\": \"description\", \"type\": \"text\"},\n", + " {\n", + " \"name\": \"embedding\",\n", + " \"type\": \"vector\",\n", + " \"attrs\": {\n", + " \"dims\": dims,\n", + " \"algorithm\": \"flat\",\n", + " \"datatype\": \"float32\",\n", + " \"distance_metric\": \"cosine\"\n", + " }\n", + " }\n", + " ]\n", + "}\n", + "\n", + "# Create and populate FLAT index\n", + "print(\"Creating sample FLAT index...\")\n", + "flat_index = SearchIndex.from_dict(flat_schema, redis_url=REDIS_URL)\n", + "flat_index.create(overwrite=True)\n", + "print(f\"โœ… Created FLAT index: {flat_index.name}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "Generate embeddings for movie descriptions\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ”„ Generating embeddings for movie descriptions...\n", + "๐Ÿš€ Using RedisVL HFTextVectorizer...\n", + "12:00:07 sentence_transformers.SentenceTransformer INFO Use pytorch device_name: mps\n", + "12:00:07 sentence_transformers.SentenceTransformer INFO Load pretrained SentenceTransformer: sentence-transformers/all-mpnet-base-v2\n", + "โœ… Generated 20 real embeddings using RedisVL HFTextVectorizer\n" + ] + } + ], + "source": [ + "# Generate embeddings using RedisVL vectorizers\n", + "print(\"๐Ÿ”„ Generating embeddings for movie descriptions...\")\n", + "embedding_model=\"sentence-transformers/all-mpnet-base-v2\"\n", + "descriptions = [movie['description'] for movie in movies_data]\n", + "\n", + "# Use RedisVL HFTextVectorizer\n", + "print(\"๐Ÿš€ Using RedisVL HFTextVectorizer...\")\n", + "vectorizer = HFTextVectorizer(\n", + " model=embedding_model # 768 dimensions\n", + ")\n", + "\n", + "# Generate embeddings using RedisVL vectorizer\n", + "embeddings = vectorizer.embed_many(descriptions)\n", + "embeddings = np.array(embeddings, dtype=np.float32)\n", + "\n", + "print(f\"โœ… Generated {len(embeddings)} real embeddings using RedisVL HFTextVectorizer\")\n", + "\n", + "# Prepare data for loading\n", + "sample_data = []\n", + "for i, movie in enumerate(movies_data):\n", + " sample_data.append({\n", + " 'movie_id': str(movie['id']),\n", + " 'title': movie['title'],\n", + " 'genre': movie['genre'],\n", + " 'rating': movie['rating'],\n", + " 'description': movie['description'],\n", + " 'embedding': array_to_buffer(embeddings[i].astype(np.float32), dtype='float32')\n", + " })" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ“ฅ Loading data into FLAT index...\n", + " Loaded 20/20 documents\n", + "Waiting for indexing to complete...\n", + "\n", + "โœ… FLAT index loaded with 40 documents\n", + "Index size: 3.0174942016601563 MB\n" + ] + } + ], + "source": [ + "# Load data into FLAT index\n", + "print(\"๐Ÿ“ฅ Loading data into FLAT index...\")\n", + "batch_size = 100 # Process in batches\n", + "\n", + "for i in range(0, len(sample_data), batch_size):\n", + " batch = sample_data[i:i+batch_size]\n", + " flat_index.load(batch)\n", + " print(f\" Loaded {min(i+batch_size, len(sample_data))}/{len(sample_data)} documents\")\n", + "\n", + "# Wait for indexing to complete\n", + "print(\"Waiting for indexing to complete...\")\n", + "time.sleep(3)\n", + "\n", + "flat_info = flat_index.info()\n", + "print(f\"\\nโœ… FLAT index loaded with {flat_info['num_docs']} documents\")\n", + "print(f\"Index size: {flat_info.get('vector_index_sz_mb', 'N/A')} MB\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 3: Get Compression Recommendation\n", + "\n", + "The CompressionAdvisor analyzes your vector dimensions and provides optimal compression settings for SVS-VAMANA vector indices. It eliminates the guesswork from parameter tuning by providing intelligent recommendations based on your vector characteristics and performance priorities.\n", + "\n", + "## Configuration Strategy\n", + "**High-Dimensional Vectors (โ‰ฅ1024 dims)**: Uses **LeanVec4x8** compression with dimensionality reduction. Memory priority reduces dimensions by 50%, speed priority by\n", + "25%, balanced by 50%. Achieves 60-80% memory savings.\n", + "\n", + "**Lower-Dimensional Vectors (<1024 dims)**: Uses **LVQ compression** without dimensionality reduction. Memory priority uses LVQ4 (4 bits), speed uses LVQ4x8 (12 bits),\n", + "balanced uses LVQ4x4 (8 bits). Achieves 60-87% memory savings.\n", + "\n", + "**Our Configuration (768 dims)**: Will use **LVQ compression** as we're below the 1024 dimension threshold. This provides excellent compression without dimensionality reduction.\n", + "\n", + "## Available Compression Types\n", + "- **LVQ4/LVQ4x4/LVQ4x8**: 4/8/12 bits per dimension\n", + "- **LeanVec4x8/LeanVec8x8**: 12/16 bits + dimensionality reduction for high-dim vectors\n" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ” Analyzing compression options...\n", + "\n", + "MEMORY priority:\n", + " Algorithm: svs-vamana\n", + " Compression: LVQ4\n", + " Datatype: float32\n", + "\n", + "BALANCED priority:\n", + " Algorithm: svs-vamana\n", + " Compression: LVQ4x4\n", + " Datatype: float32\n", + "\n", + "PERFORMANCE priority:\n", + " Algorithm: svs-vamana\n", + " Compression: LVQ4x4\n", + " Datatype: float32\n", + "\n", + "๐Ÿ“‹ Selected configuration: LVQ4 with float32\n", + "Expected memory savings: Significant for 768-dimensional vectors\n" + ] + } + ], + "source": [ + "# Get compression recommendation\n", + "print(\"๐Ÿ” Analyzing compression options...\")\n", + "print()\n", + "\n", + "# Try different priorities to show options\n", + "priorities = [\"memory\", \"balanced\", \"performance\"]\n", + "configs = {}\n", + "\n", + "for priority in priorities:\n", + " config = CompressionAdvisor.recommend(dims=dims, priority=priority)\n", + " configs[priority] = config\n", + " print(f\"{priority.upper()} priority:\")\n", + " print(f\" Algorithm: {config.algorithm}\")\n", + " print(f\" Compression: {config.compression if hasattr(config, 'compression') else 'None'}\")\n", + " print(f\" Datatype: {config.datatype}\")\n", + " if hasattr(config, 'reduce') and config.reduce:\n", + " reduction = ((dims - config.reduce) / dims) * 100\n", + " print(f\" Dimensionality: {dims} โ†’ {config.reduce} ({reduction:.1f}% reduction)\")\n", + " print()\n", + "\n", + "# Select memory-optimized configuration for migration\n", + "selected_config = configs[\"memory\"]\n", + "print(f\"๐Ÿ“‹ Selected configuration: {selected_config.compression if hasattr(selected_config, 'compression') else 'None'} with {selected_config.datatype}\")\n", + "print(f\"Expected memory savings: Significant for {dims}-dimensional vectors\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 4: Create SVS-VAMANA Index\n", + "\n", + "Now we'll create the new SVS-VAMANA index with the recommended compression settings." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Creating SVS-VAMANA index with compression...\n", + "12:01:03 redisvl.index.index INFO Index already exists, overwriting.\n", + "โœ… Created SVS-VAMANA index: migration_demo_svs\n", + "Compression: LVQ4\n", + "Datatype: float32\n" + ] + } + ], + "source": [ + "# Fallback configuration if not defined (for CI/CD compatibility)\n", + "if 'selected_config' not in locals():\n", + " from redisvl.utils import CompressionAdvisor\n", + " selected_config = CompressionAdvisor.recommend(dims=dims, priority=\"memory\")\n", + "\n", + "# Create SVS-VAMANA schema with compression\n", + "svs_schema = {\n", + " \"index\": {\n", + " \"name\": \"migration_demo_svs\",\n", + " \"prefix\": \"demo:svs:\",\n", + " },\n", + " \"fields\": [\n", + " {\"name\": \"movie_id\", \"type\": \"tag\"},\n", + " {\"name\": \"title\", \"type\": \"text\"},\n", + " {\"name\": \"genre\", \"type\": \"tag\"},\n", + " {\"name\": \"rating\", \"type\": \"numeric\"},\n", + " {\"name\": \"description\", \"type\": \"text\"},\n", + " {\n", + " \"name\": \"embedding\",\n", + " \"type\": \"vector\",\n", + " \"attrs\": {\n", + " \"dims\": selected_config.reduce if (hasattr(selected_config, 'reduce') and selected_config.reduce is not None) else dims,\n", + " \"algorithm\": \"svs-vamana\",\n", + " \"datatype\": selected_config.datatype,\n", + " \"distance_metric\": \"cosine\"\n", + " # Note: Don't include the full selected_config to avoid dims/reduce conflict\n", + " }\n", + " }\n", + " ]\n", + "}\n", + "\n", + "print(\"Creating SVS-VAMANA index with compression...\")\n", + "svs_index = SearchIndex.from_dict(svs_schema, redis_url=REDIS_URL)\n", + "svs_index.create(overwrite=True)\n", + "print(f\"โœ… Created SVS-VAMANA index: {svs_index.name}\")\n", + "print(f\"Compression: {selected_config.compression if hasattr(selected_config, 'compression') else 'None'}\")\n", + "print(f\"Datatype: {selected_config.datatype}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 5: Migrate Data\n", + "\n", + "Extract data from the original index and load it into the SVS-VAMANA index with compression applied." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ”„ Migrating data to SVS-VAMANA...\n", + "Target dimensions: 768 (from 768)\n", + "Target datatype: float32\n" + ] + } + ], + "source": [ + "print(\"๐Ÿ”„ Migrating data to SVS-VAMANA...\")\n", + "\n", + "# Fallback configuration if not defined (for CI/CD compatibility)\n", + "if 'selected_config' not in locals():\n", + " from redisvl.utils import CompressionAdvisor\n", + " selected_config = CompressionAdvisor.recommend(dims=dims, priority=\"memory\")\n", + "\n", + "# Determine target vector dimensions (may be reduced by LeanVec)\n", + "# Use reduce if it exists and is not None, otherwise use original dims\n", + "target_dims = selected_config.reduce if (hasattr(selected_config, 'reduce') and selected_config.reduce is not None) else dims\n", + "target_dtype = selected_config.datatype\n", + "\n", + "print(f\"Target dimensions: {target_dims} (from {dims})\")\n", + "print(f\"Target datatype: {target_dtype}\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Extracting data from original index...\n", + "Found 40 documents to migrate\n", + "Prepared 40 documents for migration\n" + ] + } + ], + "source": [ + "# Extract data from FLAT index\n", + "print(\"Extracting data from original index...\")\n", + "keys = client.keys(\"demo:flat:*\")\n", + "print(f\"Found {len(keys)} documents to migrate\")\n", + "\n", + "# Process and transform data for SVS index\n", + "svs_data = []\n", + "for i, key in enumerate(keys):\n", + " doc_data = client.hgetall(key)\n", + " \n", + " if b'embedding' in doc_data:\n", + " # Extract original vector\n", + " original_vector = np.array(buffer_to_array(doc_data[b'embedding'], dtype='float32'))\n", + " \n", + " # Apply dimensionality reduction if needed (LeanVec)\n", + " if target_dims < dims:\n", + " vector = original_vector[:target_dims]\n", + " else:\n", + " vector = original_vector\n", + " \n", + " # Convert to target datatype\n", + " if target_dtype == 'float16':\n", + " vector = vector.astype(np.float16)\n", + " \n", + " svs_data.append({\n", + " \"movie_id\": doc_data[b'movie_id'].decode(),\n", + " \"title\": doc_data[b'title'].decode(),\n", + " \"genre\": doc_data[b'genre'].decode(),\n", + " \"rating\": int(doc_data[b'rating'].decode()),\n", + " \"description\": doc_data[b'description'].decode(),\n", + " \"embedding\": array_to_buffer(vector, dtype=target_dtype)\n", + " })\n", + " \n", + " if (i + 1) % 500 == 0:\n", + " print(f\" Processed {i + 1}/{len(keys)} documents\")\n", + "\n", + "print(f\"Prepared {len(svs_data)} documents for migration\")" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loading data into SVS-VAMANA index...\n", + " Migrated 40/40 documents\n", + "Waiting for indexing to complete...\n", + "\n", + "โœ… Migration complete! SVS index has 60 documents\n" + ] + } + ], + "source": [ + "# Load data into SVS index\n", + "print(\"Loading data into SVS-VAMANA index...\")\n", + "batch_size = 100 # Define batch size for migration\n", + "\n", + "if len(svs_data) > 0:\n", + " for i in range(0, len(svs_data), batch_size):\n", + " batch = svs_data[i:i+batch_size]\n", + " svs_index.load(batch)\n", + " print(f\" Migrated {min(i+batch_size, len(svs_data))}/{len(svs_data)} documents\")\n", + "\n", + " # Wait for indexing to complete\n", + " print(\"Waiting for indexing to complete...\")\n", + " time.sleep(5)\n", + "\n", + " svs_info = svs_index.info()\n", + " print(f\"\\nโœ… Migration complete! SVS index has {svs_info['num_docs']} documents\")\n", + "else:\n", + " print(\"โš ๏ธ No data to migrate. Make sure the FLAT index was populated first.\")\n", + " print(\" Run the previous cells to load data into the FLAT index.\")\n", + " svs_info = svs_index.info()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 6: Compare Memory Usage\n", + "\n", + "Let's analyze the memory savings achieved through compression. This is just an example on the small sample data. Use a larger dataset before deciding." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ“Š Memory Usage Comparison\n", + "========================================\n", + "Original FLAT index: 3.02 MB\n", + "SVS-VAMANA index: 3.02 MB\n", + "\n", + "๐Ÿ’ฐ Memory savings: -0.1%\n", + "Absolute reduction: -0.00 MB\n" + ] + } + ], + "source": [ + "# Helper function to extract memory info\n", + "def get_memory_mb(index_info):\n", + " \"\"\"Extract memory usage in MB from index info\"\"\"\n", + " memory = index_info.get('vector_index_sz_mb', 0)\n", + " if isinstance(memory, str):\n", + " try:\n", + " return float(memory)\n", + " except ValueError:\n", + " return 0.0\n", + " return float(memory)\n", + "\n", + "# Get memory usage\n", + "flat_memory = get_memory_mb(flat_info)\n", + "svs_memory = get_memory_mb(svs_info)\n", + "\n", + "print(\n", + " \"๐Ÿ“Š Memory Usage Comparison\",\n", + " \"=\" * 40,\n", + " f\"Original FLAT index: {flat_memory:.2f} MB\",\n", + " f\"SVS-VAMANA index: {svs_memory:.2f} MB\",\n", + " \"\",\n", + " sep=\"\\n\"\n", + ")\n", + "\n", + "if flat_memory > 0:\n", + " if svs_memory > 0:\n", + " savings = ((flat_memory - svs_memory) / flat_memory) * 100\n", + " print(\n", + " f\"๐Ÿ’ฐ Memory savings: {savings:.1f}%\",\n", + " f\"Absolute reduction: {flat_memory - svs_memory:.2f} MB\",\n", + " sep=\"\\n\"\n", + " )\n", + " else:\n", + " print(\"โณ SVS index still indexing - memory comparison pending\")\n", + "else:\n", + " print(\"โš ๏ธ Memory information not available\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 7: Validate Search Quality\n", + "\n", + "Compare search quality and performance between FLAT and SVS-VAMANA indices." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ“Š Search Quality Comparison\n", + "========================================\n", + "Recall@5: 0.667 (66.7%)\n", + "Recall@10: 0.800 (80.0%)\n", + "\n", + "โฑ๏ธ Performance Comparison:\n", + "FLAT query time: 0.018s (1.8ms per query)\n", + "SVS-VAMANA query time: 0.013s (1.3ms per query)\n", + "Speed difference: +28.3%\n", + "\n", + "๐ŸŽฏ Quality Assessment: ๐ŸŸ  Fair - Consider if quality requirements are flexible\n" + ] + } + ], + "source": [ + "# Calculate recall and performance metrics\n", + "def calculate_recall(reference_results, test_results, k=10):\n", + " \"\"\"Calculate recall@k between two result sets\"\"\"\n", + " if not reference_results or not test_results:\n", + " return 0.0\n", + " \n", + " ref_ids = set([doc['movie_id'] for doc in reference_results[:k]])\n", + " test_ids = set([doc['movie_id'] for doc in test_results[:k]])\n", + " \n", + " if not ref_ids:\n", + " return 0.0\n", + " \n", + " return len(ref_ids.intersection(test_ids)) / len(ref_ids)\n", + "\n", + "# Create test queries\n", + "num_test_queries = 10\n", + "test_queries = []\n", + "\n", + "for i in range(num_test_queries):\n", + " query_vec = np.random.random(dims).astype(np.float32)\n", + " query_vec = query_vec / np.linalg.norm(query_vec)\n", + " test_queries.append(query_vec)\n", + "\n", + "# Test FLAT index (ground truth)\n", + "flat_results_list = []\n", + "flat_start = time.time()\n", + "\n", + "for query_vec in test_queries:\n", + " query = VectorQuery(\n", + " vector=query_vec,\n", + " vector_field_name=\"embedding\",\n", + " return_fields=[\"movie_id\", \"title\", \"genre\"],\n", + " dtype=\"float32\",\n", + " num_results=10\n", + " )\n", + " results = flat_index.query(query)\n", + " flat_results_list.append(results)\n", + "\n", + "flat_time = time.time() - flat_start\n", + "\n", + "# Test SVS-VAMANA index\n", + "svs_results_list = []\n", + "svs_start = time.time()\n", + "\n", + "for query_vec in test_queries:\n", + " # Adjust query vector for SVS index (handle dimensionality reduction)\n", + " if target_dims < dims:\n", + " svs_query_vec = query_vec[:target_dims]\n", + " else:\n", + " svs_query_vec = query_vec\n", + " \n", + " if target_dtype == 'float16':\n", + " svs_query_vec = svs_query_vec.astype(np.float16)\n", + " \n", + " query = VectorQuery(\n", + " vector=svs_query_vec,\n", + " vector_field_name=\"embedding\",\n", + " return_fields=[\"movie_id\", \"title\", \"genre\"],\n", + " dtype=target_dtype,\n", + " num_results=10\n", + " )\n", + " results = svs_index.query(query)\n", + " svs_results_list.append(results)\n", + "\n", + "svs_time = time.time() - svs_start\n", + "\n", + "# Calculate recall metrics\n", + "recall_at_5 = np.mean([calculate_recall(flat_res, svs_res, k=5) \n", + " for flat_res, svs_res in zip(flat_results_list, svs_results_list)])\n", + "recall_at_10 = np.mean([calculate_recall(flat_res, svs_res, k=10) \n", + " for flat_res, svs_res in zip(flat_results_list, svs_results_list)])\n", + "\n", + "print(\n", + " \"๐Ÿ“Š Search Quality Comparison\",\n", + " \"=\" * 40,\n", + " \"FLAT (baseline): 100% recall (brute-force exact search)\",\n", + " f\"SVS-VAMANA Recall@5: {recall_at_5*100:.1f}% (vs FLAT baseline)\",\n", + " f\"SVS-VAMANA Recall@10: {recall_at_10*100:.1f}% (vs FLAT baseline)\",\n", + " \"\",\n", + " \"โฑ๏ธ Performance Comparison:\",\n", + " f\"FLAT query time: {flat_time:.3f}s ({flat_time/num_test_queries*1000:.1f}ms per query)\",\n", + " f\"SVS-VAMANA query time: {svs_time:.3f}s ({svs_time/num_test_queries*1000:.1f}ms per query)\",\n", + " f\"Speed difference: {((flat_time - svs_time) / flat_time * 100):+.1f}%\",\n", + " sep=\"\\n\"\n", + ")\n", + "\n", + "# Quality assessment\n", + "if recall_at_10 >= 0.95:\n", + " quality_assessment = \"๐ŸŸข Excellent - Minimal quality loss\"\n", + "elif recall_at_10 >= 0.90:\n", + " quality_assessment = \"๐ŸŸก Good - Acceptable quality for most applications\"\n", + "elif recall_at_10 >= 0.80:\n", + " quality_assessment = \"๐ŸŸ  Fair - Consider if quality requirements are flexible\"\n", + "else:\n", + " quality_assessment = \"๐Ÿ”ด Poor - Migration not recommended\"\n", + "\n", + "print(f\"\\n๐ŸŽฏ Quality Assessment: {quality_assessment}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 8: Migration Decision Framework\n", + "\n", + "Analyze the migration results and provide a recommendation based on memory savings and search quality." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿค” Migration Decision Analysis\n", + "========================================\n", + "\n", + "๐Ÿ“Š Criteria Evaluation:\n", + "Memory savings: -0.1% โŒ (threshold: 5%)\n", + "Search quality: 0.800 โŒ (threshold: 0.85)\n", + "\n", + "๐ŸŽฏ Migration Recommendation: ๐Ÿ”ด NOT RECOMMENDED\n", + "๐Ÿ’ญ Reasoning: Insufficient memory savings and/or poor search quality. Consider alternative optimization strategies.\n" + ] + } + ], + "source": [ + "# Migration decision logic\n", + "memory_savings_threshold = 5 # Minimum % memory savings\n", + "recall_threshold = 0.85 # Minimum 85% recall@10\n", + "\n", + "memory_savings_pct = ((flat_memory - svs_memory) / flat_memory * 100) if flat_memory > 0 and svs_memory > 0 else 0\n", + "meets_memory_threshold = memory_savings_pct >= memory_savings_threshold\n", + "meets_quality_threshold = recall_at_10 >= recall_threshold\n", + "\n", + "print(\n", + " \"๐Ÿค” Migration Decision Analysis\",\n", + " \"=\" * 40,\n", + " \"\",\n", + " \"๐Ÿ“Š Criteria Evaluation:\",\n", + " f\"Memory savings: {memory_savings_pct:.1f}% {'โœ…' if meets_memory_threshold else 'โŒ'} (threshold: {memory_savings_threshold}%)\",\n", + " f\"Search quality: {recall_at_10:.3f} {'โœ…' if meets_quality_threshold else 'โŒ'} (threshold: {recall_threshold})\",\n", + " \"\",\n", + " sep=\"\\n\"\n", + ")\n", + "\n", + "if meets_memory_threshold and meets_quality_threshold:\n", + " recommendation = \"๐ŸŸข RECOMMENDED\"\n", + " reasoning = \"Migration provides significant memory savings while maintaining good search quality.\"\n", + "elif meets_memory_threshold and not meets_quality_threshold:\n", + " recommendation = \"๐ŸŸก CONDITIONAL\"\n", + " reasoning = \"Good memory savings but reduced search quality. Consider if your application can tolerate lower recall.\"\n", + "elif not meets_memory_threshold and meets_quality_threshold:\n", + " recommendation = \"๐ŸŸ  LIMITED BENEFIT\"\n", + " reasoning = \"Search quality is maintained but memory savings are minimal. Migration may not be worth the effort.\"\n", + "else:\n", + " recommendation = \"๐Ÿ”ด NOT RECOMMENDED\"\n", + " reasoning = \"Insufficient memory savings and/or poor search quality. Consider alternative optimization strategies.\"\n", + "\n", + "print(\n", + " f\"๐ŸŽฏ Migration Recommendation: {recommendation}\",\n", + " f\"๐Ÿ’ญ Reasoning: {reasoning}\",\n", + " sep=\"\\n\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 9: Cleanup\n", + "\n", + "Clean up the demonstration indices." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿงน Cleaning up demonstration indices...\n", + "โœ… Deleted FLAT demonstration index\n", + "โœ… Deleted SVS-VAMANA demonstration index\n", + "\n", + "๐ŸŽ‰ Migration demonstration complete!\n", + "\n", + "Next steps:\n", + "1. Apply learnings to your production data\n", + "2. Test with your actual query patterns\n", + "3. Monitor performance in your environment\n", + "4. Consider gradual rollout strategy\n" + ] + } + ], + "source": [ + "print(\"๐Ÿงน Cleaning up demonstration indices...\")\n", + "\n", + "# Clean up FLAT index\n", + "try:\n", + " flat_index.delete(drop=True)\n", + " print(\"โœ… Deleted FLAT demonstration index\")\n", + "except Exception as e:\n", + " print(f\"โš ๏ธ Failed to delete FLAT index: {e}\")\n", + "\n", + "# Clean up SVS index\n", + "try:\n", + " svs_index.delete(drop=True)\n", + " print(\"โœ… Deleted SVS-VAMANA demonstration index\")\n", + "except Exception as e:\n", + " print(f\"โš ๏ธ Failed to delete SVS index: {e}\")\n", + "\n", + "print(\n", + " \"\\n๐ŸŽ‰ Migration demonstration complete!\",\n", + " \"\\nNext steps:\",\n", + " \"1. Apply learnings to your production data\",\n", + " \"2. Test with your actual query patterns\",\n", + " \"3. Monitor performance in your environment\",\n", + " \"4. Consider gradual rollout strategy\",\n", + " sep=\"\\n\"\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.6" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}