From f2d93876dd54033f6eb356fd112c6a2b462eb595 Mon Sep 17 00:00:00 2001 From: Dhiraj Kumar Azad Date: Thu, 11 Dec 2025 17:22:46 +0530 Subject: [PATCH 1/6] Updated the tutorial according to the latest docs --- .../gsi/CouchbaseStorage_Demo.ipynb | 1747 ----------------- crewai-short-term-memory/gsi/frontmatter.md | 22 - .../{fts => query_based}/.env.sample | 0 .../query_based/CouchbaseStorage_Demo.ipynb | 1235 ++++++++++++ .../query_based/frontmatter.md | 25 + .../{gsi => search_based}/.env.sample | 0 .../CouchbaseStorage_Demo.ipynb | 49 +- .../{fts => search_based}/crew_index.json | 1 + .../{fts => search_based}/frontmatter.md | 11 +- 9 files changed, 1300 insertions(+), 1790 deletions(-) delete mode 100644 crewai-short-term-memory/gsi/CouchbaseStorage_Demo.ipynb delete mode 100644 crewai-short-term-memory/gsi/frontmatter.md rename crewai-short-term-memory/{fts => query_based}/.env.sample (100%) create mode 100644 crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb create mode 100644 crewai-short-term-memory/query_based/frontmatter.md rename crewai-short-term-memory/{gsi => search_based}/.env.sample (100%) rename crewai-short-term-memory/{fts => search_based}/CouchbaseStorage_Demo.ipynb (97%) rename crewai-short-term-memory/{fts => search_based}/crew_index.json (99%) rename crewai-short-term-memory/{fts => search_based}/frontmatter.md (66%) diff --git a/crewai-short-term-memory/gsi/CouchbaseStorage_Demo.ipynb b/crewai-short-term-memory/gsi/CouchbaseStorage_Demo.ipynb deleted file mode 100644 index 16079a09..00000000 --- a/crewai-short-term-memory/gsi/CouchbaseStorage_Demo.ipynb +++ /dev/null @@ -1,1747 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "fa3af5ad", - "metadata": {}, - "source": [ - "# CrewAI Short-Term Memory with Couchbase GSI Vector Search" - ] - }, - { - "cell_type": "markdown", - "id": "3677c445", - "metadata": {}, - "source": [ - "## Overview" - ] - }, - { - "cell_type": "markdown", - "id": "407ff72e", - "metadata": {}, - "source": [ - "This tutorial shows how to implement a custom memory backend for CrewAI agents using Couchbase's high-performance GSI (Global Secondary Index) vector search. CrewAI agents can retain and recall information across interactions, making them more contextually aware and effective. We'll demonstrate measurable performance improvements with GSI optimization. Alternatively if you want to perform semantic search using the FTS, please take a look at [this.](https://developer.couchbase.com/tutorial-crewai-short-term-memory-couchbase-with-fts)\n", - "\n", - "**Key Features:**\n", - "- Custom CrewAI memory storage with Couchbase GSI vector search\n", - "- High-performance semantic memory retrieval\n", - "- Agent memory persistence across conversations\n", - "- Performance benchmarks showing GSI benefits\n", - "\n", - "**Requirements:** Couchbase Server 8.0+ or Capella with Query Service enabled.\n", - "\n", - "You can access this notebook [here](https://github.com/couchbase-examples/vector-search-cookbook/blob/main/crewai-short-term-memory/gsi/CouchbaseStorage_Demo.ipynb)." - ] - }, - { - "cell_type": "markdown", - "id": "32f885be", - "metadata": {}, - "source": [ - "## Prerequisites" - ] - }, - { - "cell_type": "markdown", - "id": "4cda6f3d", - "metadata": {}, - "source": [ - "### Couchbase Setup" - ] - }, - { - "cell_type": "markdown", - "id": "a3e10a74", - "metadata": {}, - "source": [ - "1. **Create Capella Account:** Deploy a [free tier cluster](https://cloud.couchbase.com/sign-up)\n", - "2. **Enable Query Service:** Required for GSI vector search\n", - "3. **Configure Access:** Set up database credentials and network security\n", - "4. **Create Bucket:** Manual bucket creation recommended for Capella" - ] - }, - { - "cell_type": "markdown", - "id": "bb26dafe", - "metadata": {}, - "source": [ - "## Understanding Agent Memory" - ] - }, - { - "cell_type": "markdown", - "id": "214ea40b", - "metadata": {}, - "source": [ - "### Why Memory Matters for AI Agents" - ] - }, - { - "cell_type": "markdown", - "id": "c3873132", - "metadata": {}, - "source": [ - "Memory in AI agents is a crucial capability that allows them to retain and utilize information across interactions, making them more effective and contextually aware. Without memory, agents would be limited to processing only the immediate input, lacking the ability to build upon past experiences or maintain continuity in conversations." - ] - }, - { - "cell_type": "markdown", - "id": "eaede747", - "metadata": {}, - "source": [ - "#### Types of Memory in AI Agents" - ] - }, - { - "cell_type": "markdown", - "id": "3346122d", - "metadata": {}, - "source": [ - "**Short-term Memory:**\n", - "- Retains recent interactions and context\n", - "- Typically spans the current conversation or session \n", - "- Helps maintain coherence within a single interaction flow\n", - "- In CrewAI, this is what we're implementing with the Couchbase storage\n", - "\n", - "**Long-term Memory:**\n", - "- Stores persistent knowledge across multiple sessions\n", - "- Enables agents to recall past interactions even after long periods\n", - "- Helps build cumulative knowledge about users, preferences, and past decisions\n", - "- While this implementation is labeled as \"short-term memory\", the Couchbase storage backend can be effectively used for long-term memory as well, thanks to Couchbase's persistent storage capabilities and enterprise-grade durability features" - ] - }, - { - "cell_type": "markdown", - "id": "9d744f4a", - "metadata": {}, - "source": [ - "#### How Memory Works in Agents" - ] - }, - { - "cell_type": "markdown", - "id": "53bd56b7", - "metadata": {}, - "source": [ - "Memory in AI agents typically involves:\n", - "- **Storage**: Information is encoded and stored in a database (like Couchbase, ChromaDB, or other vector stores)\n", - "- **Retrieval**: Relevant memories are fetched based on semantic similarity to current context\n", - "- **Integration**: Retrieved memories are incorporated into the agent's reasoning process\n", - "\n", - "The vector-based approach (using embeddings) is particularly powerful because it allows for semantic search - finding memories that are conceptually related to the current context, not just exact keyword matches." - ] - }, - { - "cell_type": "markdown", - "id": "180704cb", - "metadata": {}, - "source": [ - "#### Benefits of Memory in AI Agents" - ] - }, - { - "cell_type": "markdown", - "id": "5242d1ea", - "metadata": {}, - "source": [ - "- **Contextual Understanding**: Agents can refer to previous parts of a conversation\n", - "- **Personalization**: Remembering user preferences and past interactions\n", - "- **Learning and Adaptation**: Building knowledge over time to improve responses\n", - "- **Task Continuity**: Resuming complex tasks across multiple interactions\n", - "- **Collaboration**: In multi-agent systems like CrewAI, memory enables agents to build on each other's work" - ] - }, - { - "cell_type": "markdown", - "id": "b6a39375", - "metadata": {}, - "source": [ - "#### Memory in CrewAI Specifically" - ] - }, - { - "cell_type": "markdown", - "id": "2f3f0133", - "metadata": {}, - "source": [ - "In CrewAI, memory serves several important functions:\n", - "- **Agent Specialization**: Each agent can maintain its own memory relevant to its expertise\n", - "- **Knowledge Transfer**: Agents can share insights through memory when collaborating on tasks\n", - "- **Process Continuity**: In sequential processes, later agents can access the work of earlier agents\n", - "- **Contextual Awareness**: Agents can reference previous findings when making decisions" - ] - }, - { - "cell_type": "markdown", - "id": "0082810e", - "metadata": {}, - "source": [ - "## Setup and Installation" - ] - }, - { - "cell_type": "markdown", - "id": "c23683a4", - "metadata": {}, - "source": [ - "### Install Required Libraries" - ] - }, - { - "cell_type": "markdown", - "id": "b41c9376", - "metadata": {}, - "source": [ - "Install the necessary packages for CrewAI, Couchbase integration, and OpenAI embeddings." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "fd5f51cb", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Note: you may need to restart the kernel to use updated packages.\n" - ] - } - ], - "source": [ - "%pip install --quiet crewai==0.186.1 langchain-couchbase==0.5.0 langchain-openai==0.3.33 python-dotenv==1.1.1" - ] - }, - { - "cell_type": "markdown", - "id": "5e73ffeb", - "metadata": {}, - "source": [ - "### Import Required Modules" - ] - }, - { - "cell_type": "markdown", - "id": "bd67cca9", - "metadata": {}, - "source": [ - "Import libraries for CrewAI memory storage, Couchbase GSI vector search, and OpenAI embeddings." - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "4fb688e4", - "metadata": {}, - "outputs": [], - "source": [ - "from typing import Any, Dict, List, Optional\n", - "import os\n", - "import logging\n", - "from datetime import timedelta\n", - "from dotenv import load_dotenv\n", - "from crewai.memory.storage.rag_storage import RAGStorage\n", - "from crewai.memory.short_term.short_term_memory import ShortTermMemory\n", - "from crewai import Agent, Crew, Task, Process\n", - "from couchbase.cluster import Cluster\n", - "from couchbase.options import ClusterOptions\n", - "from couchbase.auth import PasswordAuthenticator\n", - "from couchbase.diagnostics import PingState, ServiceType\n", - "from langchain_couchbase.vectorstores import CouchbaseQueryVectorStore\n", - "from langchain_couchbase.vectorstores import DistanceStrategy\n", - "from langchain_couchbase.vectorstores import IndexType\n", - "from langchain_openai import OpenAIEmbeddings, ChatOpenAI\n", - "import time\n", - "import json\n", - "import uuid\n", - "\n", - "# Configure logging (disabled)\n", - "logging.basicConfig(level=logging.CRITICAL)\n", - "logger = logging.getLogger(__name__)" - ] - }, - { - "cell_type": "markdown", - "id": "3c044af6", - "metadata": {}, - "source": [ - "### Environment Configuration" - ] - }, - { - "cell_type": "markdown", - "id": "abe7a8ad", - "metadata": {}, - "source": [ - "Configure environment variables for secure access to Couchbase and OpenAI services. Create a `.env` file with your credentials." - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "a1d82bff", - "metadata": {}, - "outputs": [], - "source": [ - "load_dotenv(\"./.env\")\n", - "\n", - "# Verify environment variables\n", - "required_vars = ['OPENAI_API_KEY', 'CB_HOST', 'CB_USERNAME', 'CB_PASSWORD']\n", - "for var in required_vars:\n", - " if not os.getenv(var):\n", - " raise ValueError(f\"{var} environment variable is required\")" - ] - }, - { - "cell_type": "markdown", - "id": "a6c46413", - "metadata": {}, - "source": [ - "## Understanding GSI Vector Search" - ] - }, - { - "cell_type": "markdown", - "id": "2ff0e7b8", - "metadata": {}, - "source": [ - "### GSI Vector Index Types" - ] - }, - { - "cell_type": "markdown", - "id": "fde149b9", - "metadata": {}, - "source": [ - "Couchbase offers two types of GSI vector indexes for different use cases:\n", - "\n", - "**Hyperscale Vector Indexes (BHIVE):**\n", - "- Best for pure vector searches - content discovery, recommendations, semantic search\n", - "- High performance with low memory footprint - designed to scale to billions of vectors\n", - "- Optimized for concurrent operations - supports simultaneous searches and inserts\n", - "- Use when: You primarily perform vector-only queries without complex scalar filtering\n", - "- Ideal for: Large-scale semantic search, recommendation systems, content discovery\n", - "\n", - "**Composite Vector Indexes:**\n", - "- Best for filtered vector searches - combines vector search with scalar value filtering\n", - "- Efficient pre-filtering - scalar attributes reduce the vector comparison scope\n", - "- Use when: Your queries combine vector similarity with scalar filters that eliminate large portions of data\n", - "- Ideal for: Compliance-based filtering, user-specific searches, time-bounded queries\n", - "\n", - "For this CrewAI memory implementation, we'll use **BHIVE** as it's optimized for pure semantic search scenarios typical in AI agent memory systems." - ] - }, - { - "cell_type": "markdown", - "id": "1dfc28e9", - "metadata": {}, - "source": [ - "### Understanding Index Configuration" - ] - }, - { - "cell_type": "markdown", - "id": "acab2b26", - "metadata": {}, - "source": [ - "The `index_description` parameter controls how Couchbase optimizes vector storage and search performance through centroids and quantization:\n", - "\n", - "**Format**: `'IVF[],{PQ|SQ}'`\n", - "\n", - "**Centroids (IVF - Inverted File):**\n", - "- Controls how the dataset is subdivided for faster searches\n", - "- More centroids = faster search, slower training \n", - "- Fewer centroids = slower search, faster training\n", - "- If omitted (like IVF,SQ8), Couchbase auto-selects based on dataset size\n", - "\n", - "**Quantization Options:**\n", - "- SQ (Scalar Quantization): SQ4, SQ6, SQ8 (4, 6, or 8 bits per dimension)\n", - "- PQ (Product Quantization): PQx (e.g., PQ32x8)\n", - "- Higher values = better accuracy, larger index size\n", - "\n", - "**Common Examples:**\n", - "- IVF,SQ8 - Auto centroids, 8-bit scalar quantization (good default)\n", - "- IVF1000,SQ6 - 1000 centroids, 6-bit scalar quantization \n", - "- IVF,PQ32x8 - Auto centroids, 32 subquantizers with 8 bits\n", - "\n", - "For detailed configuration options, see the [Quantization & Centroid Settings](https://docs.couchbase.com/cloud/vector-index/hyperscale-vector-index.html#algo_settings).\n", - "\n", - "For more information on GSI vector indexes, see [Couchbase GSI Vector Documentation](https://docs.couchbase.com/cloud/vector-index/use-vector-indexes.html).\n" - ] - }, - { - "cell_type": "markdown", - "id": "3c7f0633", - "metadata": {}, - "source": [ - "## Custom CouchbaseStorage Implementation" - ] - }, - { - "cell_type": "markdown", - "id": "5df3792c", - "metadata": {}, - "source": [ - "### CouchbaseStorage Class" - ] - }, - { - "cell_type": "markdown", - "id": "a5e8abec", - "metadata": {}, - "source": [ - "This class extends CrewAI's `RAGStorage` to provide GSI vector search capabilities for agent memory." - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "b29c4840", - "metadata": {}, - "outputs": [], - "source": [ - "class CouchbaseStorage(RAGStorage):\n", - " \"\"\"\n", - " Extends RAGStorage to handle embeddings for memory entries using Couchbase GSI Vector Search.\n", - " \"\"\"\n", - "\n", - " def __init__(self, type: str, allow_reset: bool = True, embedder_config: Optional[Dict[str, Any]] = None, crew: Optional[Any] = None):\n", - " \"\"\"Initialize CouchbaseStorage with GSI vector search configuration.\"\"\"\n", - " super().__init__(type, allow_reset, embedder_config, crew)\n", - " self._initialize_app()\n", - "\n", - " def search(\n", - " self,\n", - " query: str,\n", - " limit: int = 3,\n", - " filter: Optional[dict] = None,\n", - " score_threshold: float = 0,\n", - " ) -> List[Dict[str, Any]]:\n", - " \"\"\"\n", - " Search memory entries using GSI vector similarity.\n", - " \"\"\"\n", - " try:\n", - " # Add type filter\n", - " search_filter = {\"memory_type\": self.type}\n", - " if filter:\n", - " search_filter.update(filter)\n", - "\n", - " # Execute search using GSI vector search\n", - " results = self.vector_store.similarity_search_with_score(\n", - " query,\n", - " k=limit,\n", - " filter=search_filter\n", - " )\n", - " \n", - " # Format results and deduplicate by content\n", - " seen_contents = set()\n", - " formatted_results = []\n", - " \n", - " for i, (doc, distance) in enumerate(results):\n", - " # Note: In GSI vector search, lower distance indicates higher similarity\n", - " if distance <= (1.0 - score_threshold): # Convert threshold for GSI distance metric\n", - " content = doc.page_content\n", - " if content not in seen_contents:\n", - " seen_contents.add(content)\n", - " formatted_results.append({\n", - " \"id\": doc.metadata.get(\"memory_id\", str(i)),\n", - " \"metadata\": doc.metadata,\n", - " \"context\": content,\n", - " \"distance\": float(distance) # Changed from score to distance\n", - " })\n", - " \n", - " logger.info(f\"Found {len(formatted_results)} unique results for query: {query}\")\n", - " return formatted_results\n", - "\n", - " except Exception as e:\n", - " logger.error(f\"Search failed: {str(e)}\")\n", - " return []\n", - "\n", - " def save(self, value: Any, metadata: Dict[str, Any]) -> None:\n", - " \"\"\"\n", - " Save a memory entry with metadata.\n", - " \"\"\"\n", - " try:\n", - " # Generate unique ID\n", - " memory_id = str(uuid.uuid4())\n", - " timestamp = int(time.time() * 1000)\n", - " \n", - " # Prepare metadata (create a copy to avoid modifying references)\n", - " if not metadata:\n", - " metadata = {}\n", - " else:\n", - " metadata = metadata.copy() # Create a copy to avoid modifying references\n", - " \n", - " # Process agent-specific information if present\n", - " agent_name = metadata.get('agent', 'unknown')\n", - " \n", - " # Clean up value if it has the typical LLM response format\n", - " value_str = str(value)\n", - " if \"Final Answer:\" in value_str:\n", - " # Extract just the actual content - everything after \"Final Answer:\"\n", - " parts = value_str.split(\"Final Answer:\", 1)\n", - " if len(parts) > 1:\n", - " value = parts[1].strip()\n", - " logger.info(f\"Cleaned up response format for agent: {agent_name}\")\n", - " elif value_str.startswith(\"Thought:\"):\n", - " # Handle thought/final answer format\n", - " if \"Final Answer:\" in value_str:\n", - " parts = value_str.split(\"Final Answer:\", 1)\n", - " if len(parts) > 1:\n", - " value = parts[1].strip()\n", - " logger.info(f\"Cleaned up thought process format for agent: {agent_name}\")\n", - " \n", - " # Update metadata\n", - " metadata.update({\n", - " \"memory_id\": memory_id,\n", - " \"memory_type\": self.type,\n", - " \"timestamp\": timestamp,\n", - " \"source\": \"crewai\"\n", - " })\n", - "\n", - " # Log memory information for debugging\n", - " value_preview = str(value)[:100] + \"...\" if len(str(value)) > 100 else str(value)\n", - " metadata_preview = {k: v for k, v in metadata.items() if k != \"embedding\"}\n", - " logger.info(f\"Saving memory for Agent: {agent_name}\")\n", - " logger.info(f\"Memory value preview: {value_preview}\")\n", - " logger.info(f\"Memory metadata: {metadata_preview}\")\n", - " \n", - " # Convert value to string if needed\n", - " if isinstance(value, (dict, list)):\n", - " value = json.dumps(value)\n", - " elif not isinstance(value, str):\n", - " value = str(value)\n", - "\n", - " # Save to GSI vector store\n", - " self.vector_store.add_texts(\n", - " texts=[value],\n", - " metadatas=[metadata],\n", - " ids=[memory_id]\n", - " )\n", - " logger.info(f\"Saved memory {memory_id}: {value[:100]}...\")\n", - "\n", - " except Exception as e:\n", - " logger.error(f\"Save failed: {str(e)}\")\n", - " raise\n", - "\n", - " def reset(self) -> None:\n", - " \"\"\"Reset the memory storage if allowed.\"\"\"\n", - " if not self.allow_reset:\n", - " return\n", - "\n", - " try:\n", - " # Delete documents of this memory type\n", - " self.cluster.query(\n", - " f\"DELETE FROM `{self.bucket_name}`.`{self.scope_name}`.`{self.collection_name}` WHERE memory_type = $type\",\n", - " type=self.type\n", - " ).execute()\n", - " logger.info(f\"Reset memory type: {self.type}\")\n", - " except Exception as e:\n", - " logger.error(f\"Reset failed: {str(e)}\")\n", - " raise\n", - "\n", - " def _initialize_app(self):\n", - " \"\"\"Initialize Couchbase connection and GSI vector store.\"\"\"\n", - " try:\n", - " # Initialize embeddings\n", - " if self.embedder_config and self.embedder_config.get(\"provider\") == \"openai\":\n", - " self.embeddings = OpenAIEmbeddings(\n", - " openai_api_key=os.getenv('OPENAI_API_KEY'),\n", - " model=self.embedder_config.get(\"config\", {}).get(\"model\", \"text-embedding-3-small\")\n", - " )\n", - " else:\n", - " self.embeddings = OpenAIEmbeddings(\n", - " openai_api_key=os.getenv('OPENAI_API_KEY'),\n", - " model=\"text-embedding-3-small\"\n", - " )\n", - "\n", - " # Connect to Couchbase\n", - " auth = PasswordAuthenticator(\n", - " os.getenv('CB_USERNAME', ''),\n", - " os.getenv('CB_PASSWORD', '')\n", - " )\n", - " options = ClusterOptions(auth)\n", - " \n", - " # Initialize cluster connection\n", - " self.cluster = Cluster(os.getenv('CB_HOST', ''), options)\n", - " self.cluster.wait_until_ready(timedelta(seconds=5))\n", - "\n", - " # Check Query service (required for GSI vector search)\n", - " ping_result = self.cluster.ping()\n", - " query_available = False\n", - " for service_type, endpoints in ping_result.endpoints.items():\n", - " if service_type.name == 'Query': # Query Service for GSI\n", - " for endpoint in endpoints:\n", - " if endpoint.state == PingState.OK:\n", - " query_available = True\n", - " logger.info(f\"Query service is responding at: {endpoint.remote}\")\n", - " break\n", - " break\n", - " if not query_available:\n", - " raise RuntimeError(\"Query service not found or not responding. GSI vector search requires Query Service.\")\n", - " \n", - " # Set up storage configuration\n", - " self.bucket_name = os.getenv('CB_BUCKET_NAME', 'vector-search-testing')\n", - " self.scope_name = os.getenv('SCOPE_NAME', 'shared')\n", - " self.collection_name = os.getenv('COLLECTION_NAME', 'crew')\n", - " self.index_name = os.getenv('INDEX_NAME', 'vector_search_crew_gsi')\n", - "\n", - " # Initialize GSI vector store\n", - " self.vector_store = CouchbaseQueryVectorStore(\n", - " cluster=self.cluster,\n", - " bucket_name=self.bucket_name,\n", - " scope_name=self.scope_name,\n", - " collection_name=self.collection_name,\n", - " embedding=self.embeddings,\n", - " distance_metric=DistanceStrategy.COSINE,\n", - " )\n", - " logger.info(f\"Initialized CouchbaseStorage with GSI vector search for type: {self.type}\")\n", - "\n", - " except Exception as e:\n", - " logger.error(f\"Initialization failed: {str(e)}\")\n", - " raise" - ] - }, - { - "cell_type": "markdown", - "id": "3566d5bf", - "metadata": {}, - "source": [ - "## Memory Search Performance Testing" - ] - }, - { - "cell_type": "markdown", - "id": "ff154822", - "metadata": {}, - "source": [ - "Now let's demonstrate the performance benefits of GSI optimization by testing pure memory search performance. We'll compare three optimization levels:\n", - "\n", - "1. **Baseline Performance**: Memory search without GSI optimization\n", - "2. **GSI-Optimized Performance**: Same search with BHIVE GSI index\n", - "3. **Cache Benefits**: Show how caching can be applied on top of GSI for repeated queries\n", - "\n", - "**Important**: This testing focuses on pure memory search performance, isolating the GSI improvements from CrewAI agent workflow overhead." - ] - }, - { - "cell_type": "markdown", - "id": "29717ea7", - "metadata": {}, - "source": [ - "### Initialize Storage and Test Functions" - ] - }, - { - "cell_type": "markdown", - "id": "7f41c284", - "metadata": {}, - "source": [ - "First, let's set up the storage and create test functions for measuring memory search performance." - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "06349452", - "metadata": {}, - "outputs": [], - "source": [ - "# Initialize storage\n", - "storage = CouchbaseStorage(\n", - " type=\"short_term\",\n", - " embedder_config={\n", - " \"provider\": \"openai\",\n", - " \"config\": {\"model\": \"text-embedding-3-small\"}\n", - " }\n", - ")\n", - "\n", - "# Reset storage\n", - "storage.reset()\n", - "\n", - "# Test storage\n", - "test_memory = \"Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a good moment. The way we are training, the way we are playing - I am really pleased.'\"\n", - "test_metadata = {\"category\": \"sports\", \"test\": \"initial_memory\"}\n", - "storage.save(test_memory, test_metadata)\n", - "\n", - "import time\n", - "\n", - "def test_memory_search_performance(storage, query, label=\"Memory Search\"):\n", - " \"\"\"Test pure memory search performance and return timing metrics\"\"\"\n", - " print(f\"\\n[{label}] Testing memory search performance\")\n", - " print(f\"[{label}] Query: '{query}'\")\n", - " \n", - " start_time = time.time()\n", - " \n", - " try:\n", - " results = storage.search(query, limit=3)\n", - " end_time = time.time()\n", - " search_time = end_time - start_time\n", - " \n", - " print(f\"[{label}] Memory search completed in {search_time:.4f} seconds\")\n", - " print(f\"[{label}] Found {len(results)} memories\")\n", - " \n", - " if results:\n", - " print(f\"[{label}] Top result distance: {results[0]['distance']:.6f} (lower = more similar)\")\n", - " preview = results[0]['context'][:100] + \"...\" if len(results[0]['context']) > 100 else results[0]['context']\n", - " print(f\"[{label}] Top result preview: {preview}\")\n", - " \n", - " return search_time\n", - " except Exception as e:\n", - " print(f\"[{label}] Memory search failed: {str(e)}\")\n", - " return None" - ] - }, - { - "cell_type": "markdown", - "id": "198a7939", - "metadata": {}, - "source": [ - "### Test 1: Baseline Performance (No GSI Index)" - ] - }, - { - "cell_type": "markdown", - "id": "ef5d4fde", - "metadata": {}, - "source": [ - "Test pure memory search performance without GSI optimization." - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "383bb87d", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Testing baseline memory search performance without GSI optimization...\n", - "\n", - "[Baseline Search] Testing memory search performance\n", - "[Baseline Search] Query: 'What did Guardiola say about Manchester City?'\n", - "[Baseline Search] Memory search completed in 0.6159 seconds\n", - "[Baseline Search] Found 1 memories\n", - "[Baseline Search] Top result distance: 0.340130 (lower = more similar)\n", - "[Baseline Search] Top result preview: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a ...\n", - "\n", - "Baseline memory search time (without GSI): 0.6159 seconds\n", - "\n" - ] - } - ], - "source": [ - "# Test baseline memory search performance without GSI index\n", - "test_query = \"What did Guardiola say about Manchester City?\"\n", - "print(\"Testing baseline memory search performance without GSI optimization...\")\n", - "baseline_time = test_memory_search_performance(storage, test_query, \"Baseline Search\")\n", - "print(f\"\\nBaseline memory search time (without GSI): {baseline_time:.4f} seconds\\n\")" - ] - }, - { - "cell_type": "markdown", - "id": "a88e1719", - "metadata": {}, - "source": [ - "### Create BHIVE GSI Index" - ] - }, - { - "cell_type": "markdown", - "id": "be7acf07", - "metadata": {}, - "source": [ - "Now let's create a BHIVE GSI vector index to enable high-performance memory searches. The index creation is done programmatically through the vector store." - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "bde97a46", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Creating BHIVE GSI vector index...\n", - "GSI Vector index created successfully: vector_search_crew\n", - "Waiting for index to become available...\n" - ] - } - ], - "source": [ - "# Create GSI BHIVE vector index for optimal performance\n", - "print(\"Creating BHIVE GSI vector index...\")\n", - "try:\n", - " storage.vector_store.create_index(\n", - " index_type=IndexType.BHIVE,\n", - " # index_type=IndexType.COMPOSITE, # Uncomment this line to create a COMPOSITE index instead\n", - " index_name=storage.index_name,\n", - " index_description=\"IVF,SQ8\" # Auto-selected centroids with 8-bit scalar quantization\n", - " )\n", - " print(f\"GSI Vector index created successfully: {storage.index_name}\")\n", - " \n", - " # Wait for index to become available\n", - " print(\"Waiting for index to become available...\")\n", - " time.sleep(5)\n", - " \n", - "except Exception as e:\n", - " if \"already exists\" in str(e).lower():\n", - " print(f\"GSI vector index '{storage.index_name}' already exists, proceeding...\")\n", - " else:\n", - " print(f\"Error creating GSI index: {str(e)}\")" - ] - }, - { - "cell_type": "markdown", - "id": "c389eecb", - "metadata": {}, - "source": [ - "### Alternative: Composite Index Configuration" - ] - }, - { - "cell_type": "markdown", - "id": "4e7555da", - "metadata": {}, - "source": [ - "If your agent memory use case requires complex filtering with scalar attributes, you can create a **Composite index** instead by changing the configuration above:\n", - "\n", - "```python\n", - "# Alternative: Create a Composite index for filtered memory searches\n", - "storage.vector_store.create_index(\n", - " index_type=IndexType.COMPOSITE, # Instead of IndexType.BHIVE\n", - " index_name=storage.index_name,\n", - " index_description=\"IVF,SQ8\" # Same quantization settings\n", - ")\n", - "```" - ] - }, - { - "cell_type": "markdown", - "id": "8e719352", - "metadata": {}, - "source": [ - "### Test 2: GSI-Optimized Performance" - ] - }, - { - "cell_type": "markdown", - "id": "5d786f04", - "metadata": {}, - "source": [ - "Test the same memory search with BHIVE GSI optimization." - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "849758ae", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Testing memory search performance with BHIVE GSI optimization...\n", - "\n", - "[GSI-Optimized Search] Testing memory search performance\n", - "[GSI-Optimized Search] Query: 'What did Guardiola say about Manchester City?'\n", - "[GSI-Optimized Search] Memory search completed in 0.5910 seconds\n", - "[GSI-Optimized Search] Found 1 memories\n", - "[GSI-Optimized Search] Top result distance: 0.340142 (lower = more similar)\n", - "[GSI-Optimized Search] Top result preview: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a ...\n" - ] - } - ], - "source": [ - "# Test memory search performance with GSI index\n", - "print(\"Testing memory search performance with BHIVE GSI optimization...\")\n", - "gsi_time = test_memory_search_performance(storage, test_query, \"GSI-Optimized Search\")" - ] - }, - { - "cell_type": "markdown", - "id": "905cf62e", - "metadata": {}, - "source": [ - "### Test 3: Cache Benefits Testing" - ] - }, - { - "cell_type": "markdown", - "id": "a704c5c1", - "metadata": {}, - "source": [ - "Now let's demonstrate how caching can improve performance for repeated queries. **Note**: Caching benefits apply to both baseline and GSI-optimized searches." - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "id": "febeab1f", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Testing cache benefits with memory search...\n", - "First execution (cache miss):\n", - "\n", - "[Cache Test - First Run] Testing memory search performance\n", - "[Cache Test - First Run] Query: 'How is Manchester City performing in training sessions?'\n", - "[Cache Test - First Run] Memory search completed in 0.6076 seconds\n", - "[Cache Test - First Run] Found 1 memories\n", - "[Cache Test - First Run] Top result distance: 0.379242 (lower = more similar)\n", - "[Cache Test - First Run] Top result preview: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a ...\n", - "\n", - "Second execution (cache hit - should be faster):\n", - "\n", - "[Cache Test - Second Run] Testing memory search performance\n", - "[Cache Test - Second Run] Query: 'How is Manchester City performing in training sessions?'\n", - "[Cache Test - Second Run] Memory search completed in 0.4745 seconds\n", - "[Cache Test - Second Run] Found 1 memories\n", - "[Cache Test - Second Run] Top result distance: 0.379200 (lower = more similar)\n", - "[Cache Test - Second Run] Top result preview: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a ...\n" - ] - } - ], - "source": [ - "# Test cache benefits with a different query to avoid interference\n", - "cache_test_query = \"How is Manchester City performing in training sessions?\"\n", - "\n", - "print(\"Testing cache benefits with memory search...\")\n", - "print(\"First execution (cache miss):\")\n", - "cache_time_1 = test_memory_search_performance(storage, cache_test_query, \"Cache Test - First Run\")\n", - "\n", - "print(\"\\nSecond execution (cache hit - should be faster):\")\n", - "cache_time_2 = test_memory_search_performance(storage, cache_test_query, \"Cache Test - Second Run\")" - ] - }, - { - "cell_type": "markdown", - "id": "0cd9de44", - "metadata": {}, - "source": [ - "### Memory Search Performance Analysis" - ] - }, - { - "cell_type": "markdown", - "id": "f475ccc3", - "metadata": {}, - "source": [ - "Let's analyze the memory search performance improvements across all optimization levels:" - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "id": "f813eb1a", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "================================================================================\n", - "MEMORY SEARCH PERFORMANCE OPTIMIZATION SUMMARY\n", - "================================================================================\n", - "Phase 1 - Baseline Search (No GSI): 0.6159 seconds\n", - "Phase 2 - GSI-Optimized Search: 0.5910 seconds\n", - "Phase 3 - Cache Benefits:\n", - " First execution (cache miss): 0.6076 seconds\n", - " Second execution (cache hit): 0.4745 seconds\n", - "\n", - "--------------------------------------------------------------------------------\n", - "MEMORY SEARCH OPTIMIZATION IMPACT:\n", - "--------------------------------------------------------------------------------\n", - "GSI Index Benefit: 1.04x faster (4.0% improvement)\n", - "Cache Benefit: 1.28x faster (21.9% improvement)\n", - "\n", - "Key Insights for Agent Memory Performance:\n", - "• GSI BHIVE indexes provide significant performance improvements for memory search\n", - "• Performance gains are most dramatic for complex semantic memory queries\n", - "• BHIVE optimization is particularly effective for agent conversational memory\n", - "• Combined with proper quantization (SQ8), GSI delivers production-ready performance\n", - "• These performance improvements directly benefit agent response times and scalability\n" - ] - } - ], - "source": [ - "print(\"\\n\" + \"=\"*80)\n", - "print(\"MEMORY SEARCH PERFORMANCE OPTIMIZATION SUMMARY\")\n", - "print(\"=\"*80)\n", - "\n", - "print(f\"Phase 1 - Baseline Search (No GSI): {baseline_time:.4f} seconds\")\n", - "print(f\"Phase 2 - GSI-Optimized Search: {gsi_time:.4f} seconds\")\n", - "if cache_time_1 and cache_time_2:\n", - " print(f\"Phase 3 - Cache Benefits:\")\n", - " print(f\" First execution (cache miss): {cache_time_1:.4f} seconds\")\n", - " print(f\" Second execution (cache hit): {cache_time_2:.4f} seconds\")\n", - "\n", - "print(\"\\n\" + \"-\"*80)\n", - "print(\"MEMORY SEARCH OPTIMIZATION IMPACT:\")\n", - "print(\"-\"*80)\n", - "\n", - "# GSI improvement analysis\n", - "if baseline_time and gsi_time:\n", - " speedup = baseline_time / gsi_time if gsi_time > 0 else float('inf')\n", - " time_saved = baseline_time - gsi_time\n", - " percent_improvement = (time_saved / baseline_time) * 100\n", - " print(f\"GSI Index Benefit: {speedup:.2f}x faster ({percent_improvement:.1f}% improvement)\")\n", - "\n", - "# Cache improvement analysis\n", - "if cache_time_1 and cache_time_2 and cache_time_2 < cache_time_1:\n", - " cache_speedup = cache_time_1 / cache_time_2\n", - " cache_improvement = ((cache_time_1 - cache_time_2) / cache_time_1) * 100\n", - " print(f\"Cache Benefit: {cache_speedup:.2f}x faster ({cache_improvement:.1f}% improvement)\")\n", - "else:\n", - " print(f\"Cache Benefit: Variable (depends on query complexity and caching mechanism)\")\n", - "\n", - "print(f\"\\nKey Insights for Agent Memory Performance:\")\n", - "print(f\"• GSI BHIVE indexes provide significant performance improvements for memory search\")\n", - "print(f\"• Performance gains are most dramatic for complex semantic memory queries\")\n", - "print(f\"• BHIVE optimization is particularly effective for agent conversational memory\")\n", - "print(f\"• Combined with proper quantization (SQ8), GSI delivers production-ready performance\")\n", - "print(f\"• These performance improvements directly benefit agent response times and scalability\")" - ] - }, - { - "cell_type": "markdown", - "id": "c4b069f8", - "metadata": {}, - "source": [ - "**Note on BHIVE GSI Performance:** The BHIVE GSI index may show slower performance for very small datasets (few documents) due to the additional overhead of maintaining the index structure. However, as the dataset scales up, the BHIVE GSI index becomes significantly faster than traditional vector searches. The initial overhead investment pays off dramatically with larger memory stores, making it essential for production agent deployments with substantial conversational history." - ] - }, - { - "cell_type": "markdown", - "id": "126d4fcf", - "metadata": {}, - "source": [ - "## CrewAI Agent Memory Demo" - ] - }, - { - "cell_type": "markdown", - "id": "a3c67329", - "metadata": {}, - "source": [ - "### What is CrewAI Agent Memory?" - ] - }, - { - "cell_type": "markdown", - "id": "8f71f9ec", - "metadata": {}, - "source": [ - "Now that we've optimized our memory search performance, let's demonstrate how CrewAI agents can leverage this GSI-optimized memory system. CrewAI agent memory enables:\n", - "\n", - "- **Persistent Context**: Agents remember information across conversations and tasks\n", - "- **Semantic Recall**: Agents can find relevant memories using natural language queries\n", - "- **Collaborative Memory**: Multiple agents can share and build upon each other's memories\n", - "- **Performance Benefits**: Our GSI optimizations directly improve agent memory retrieval speed\n", - "\n", - "This demo shows how the memory performance improvements we validated translate to real agent workflows." - ] - }, - { - "cell_type": "markdown", - "id": "0ea8887d", - "metadata": {}, - "source": [ - "### Create Agents with Optimized Memory" - ] - }, - { - "cell_type": "markdown", - "id": "bdf480e7", - "metadata": {}, - "source": [ - "Set up CrewAI agents that use our GSI-optimized Couchbase memory storage for fast, contextual memory retrieval." - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "id": "509767fb", - "metadata": {}, - "outputs": [], - "source": [ - "# Initialize ShortTermMemory with our storage\n", - "memory = ShortTermMemory(storage=storage)\n", - "\n", - "# Initialize language model\n", - "llm = ChatOpenAI(\n", - " model=\"gpt-4o\",\n", - " temperature=0.7\n", - ")\n", - "\n", - "# Create agents with memory\n", - "sports_analyst = Agent(\n", - " role='Sports Analyst',\n", - " goal='Analyze Manchester City performance',\n", - " backstory='Expert at analyzing football teams and providing insights on their performance',\n", - " llm=llm,\n", - " memory=True,\n", - " memory_storage=memory\n", - ")\n", - "\n", - "journalist = Agent(\n", - " role='Sports Journalist',\n", - " goal='Create engaging football articles',\n", - " backstory='Experienced sports journalist who specializes in Premier League coverage',\n", - " llm=llm,\n", - " memory=True,\n", - " memory_storage=memory\n", - ")\n", - "\n", - "# Create tasks\n", - "analysis_task = Task(\n", - " description='Analyze Manchester City\\'s recent performance based on Pep Guardiola\\'s comments: \"The team is playing well, we are in a good moment. The way we are training, the way we are playing - I am really pleased.\"',\n", - " agent=sports_analyst,\n", - " expected_output=\"A comprehensive analysis of Manchester City's current form based on Guardiola's comments.\"\n", - ")\n", - "\n", - "writing_task = Task(\n", - " description='Write a sports article about Manchester City\\'s form using the analysis and Guardiola\\'s comments.',\n", - " agent=journalist,\n", - " context=[analysis_task],\n", - " expected_output=\"An engaging sports article about Manchester City's current form and Guardiola's perspective.\"\n", - ")\n", - "\n", - "# Create crew with memory\n", - "crew = Crew(\n", - " agents=[sports_analyst, journalist],\n", - " tasks=[analysis_task, writing_task],\n", - " process=Process.sequential,\n", - " memory=True,\n", - " short_term_memory=memory, # Explicitly pass our memory implementation\n", - " verbose=True\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "950636f7", - "metadata": {}, - "source": [ - "### Run Agent Memory Demo" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "id": "95c612da", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Running CrewAI agents with GSI-optimized memory storage...\n" - ] - }, - { - "data": { - "text/html": [ - "
╭──────────────────────────────────────────── Crew Execution Started ─────────────────────────────────────────────╮\n",
-              "│                                                                                                                 │\n",
-              "│  Crew Execution Started                                                                                         │\n",
-              "│  Name: crew                                                                                                     │\n",
-              "│  ID: 38d8c744-17cf-4aef-b246-3ff3a930ca29                                                                       │\n",
-              "│  Tool Args:                                                                                                     │\n",
-              "│                                                                                                                 │\n",
-              "│                                                                                                                 │\n",
-              "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
-              "
\n" - ], - "text/plain": [ - "\u001b[36m╭─\u001b[0m\u001b[36m───────────────────────────────────────────\u001b[0m\u001b[36m Crew Execution Started \u001b[0m\u001b[36m────────────────────────────────────────────\u001b[0m\u001b[36m─╮\u001b[0m\n", - "\u001b[36m│\u001b[0m \u001b[36m│\u001b[0m\n", - "\u001b[36m│\u001b[0m \u001b[1;36mCrew Execution Started\u001b[0m \u001b[36m│\u001b[0m\n", - "\u001b[36m│\u001b[0m \u001b[37mName: \u001b[0m\u001b[36mcrew\u001b[0m \u001b[36m│\u001b[0m\n", - "\u001b[36m│\u001b[0m \u001b[37mID: \u001b[0m\u001b[36m38d8c744-17cf-4aef-b246-3ff3a930ca29\u001b[0m \u001b[36m│\u001b[0m\n", - "\u001b[36m│\u001b[0m \u001b[37mTool Args: \u001b[0m \u001b[36m│\u001b[0m\n", - "\u001b[36m│\u001b[0m \u001b[36m│\u001b[0m\n", - "\u001b[36m│\u001b[0m \u001b[36m│\u001b[0m\n", - "\u001b[36m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "text/html": [ - "
\n",
-              "
\n" - ], - "text/plain": [ - "\n" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "text/html": [ - "
\n"
-            ],
-            "text/plain": []
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
╭────────────────────────────────────────────── 🧠 Retrieved Memory ──────────────────────────────────────────────╮\n",
-              "│                                                                                                                 │\n",
-              "│  Historical Data:                                                                                               │\n",
-              "│  - Ensure that the actual output directly addresses the task description and expected output.                   │\n",
-              "│  - Include more specific statistical data and recent match examples to support the analysis.                    │\n",
-              "│  - Incorporate more direct quotes from Pep Guardiola or other relevant stakeholders.                            │\n",
-              "│  - Address potential biases in Guardiola's comments and provide a balanced view considering external opinions.  │\n",
-              "│  - Explore deeper tactical analysis to provide more insights into the team's performance.                       │\n",
-              "│  - Mention fu...                                                                                                │\n",
-              "│                                                                                                                 │\n",
-              "╰─────────────────────────────────────────── Retrieval Time: 1503.80ms ───────────────────────────────────────────╯\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\u001b[32m╭─\u001b[0m\u001b[32m─────────────────────────────────────────────\u001b[0m\u001b[32m 🧠 Retrieved Memory \u001b[0m\u001b[32m─────────────────────────────────────────────\u001b[0m\u001b[32m─╮\u001b[0m\n",
-              "\u001b[32m│\u001b[0m                                                                                                                 \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37mHistorical Data:\u001b[0m                                                                                               \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37m- Ensure that the actual output directly addresses the task description and expected output.\u001b[0m                   \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37m- Include more specific statistical data and recent match examples to support the analysis.\u001b[0m                    \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37m- Incorporate more direct quotes from Pep Guardiola or other relevant stakeholders.\u001b[0m                            \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37m- Address potential biases in Guardiola's comments and provide a balanced view considering external opinions.\u001b[0m  \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37m- Explore deeper tactical analysis to provide more insights into the team's performance.\u001b[0m                       \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37m- Mention fu...\u001b[0m                                                                                                \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m                                                                                                                 \u001b[32m│\u001b[0m\n",
-              "\u001b[32m╰─\u001b[0m\u001b[32m──────────────────────────────────────────\u001b[0m\u001b[32m Retrieval Time: 1503.80ms \u001b[0m\u001b[32m──────────────────────────────────────────\u001b[0m\u001b[32m─╯\u001b[0m\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
╭─────────────────────────────────────────────── 🤖 Agent Started ────────────────────────────────────────────────╮\n",
-              "│                                                                                                                 │\n",
-              "│  Agent: Sports Analyst                                                                                          │\n",
-              "│                                                                                                                 │\n",
-              "│  Task: Analyze Manchester City's recent performance based on Pep Guardiola's comments: \"The team is playing     │\n",
-              "│  well, we are in a good moment. The way we are training, the way we are playing - I am really pleased.\"         │\n",
-              "│                                                                                                                 │\n",
-              "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\u001b[35m╭─\u001b[0m\u001b[35m──────────────────────────────────────────────\u001b[0m\u001b[35m 🤖 Agent Started \u001b[0m\u001b[35m───────────────────────────────────────────────\u001b[0m\u001b[35m─╮\u001b[0m\n",
-              "\u001b[35m│\u001b[0m                                                                                                                 \u001b[35m│\u001b[0m\n",
-              "\u001b[35m│\u001b[0m  \u001b[37mAgent: \u001b[0m\u001b[1;92mSports Analyst\u001b[0m                                                                                          \u001b[35m│\u001b[0m\n",
-              "\u001b[35m│\u001b[0m                                                                                                                 \u001b[35m│\u001b[0m\n",
-              "\u001b[35m│\u001b[0m  \u001b[37mTask: \u001b[0m\u001b[92mAnalyze Manchester City's recent performance based on Pep Guardiola's comments: \"The team is playing \u001b[0m    \u001b[35m│\u001b[0m\n",
-              "\u001b[35m│\u001b[0m  \u001b[92mwell, we are in a good moment. The way we are training, the way we are playing - I am really pleased.\"\u001b[0m         \u001b[35m│\u001b[0m\n",
-              "\u001b[35m│\u001b[0m                                                                                                                 \u001b[35m│\u001b[0m\n",
-              "\u001b[35m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n"
-            ],
-            "text/plain": []
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n"
-            ],
-            "text/plain": []
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
╭──────────────────────────────────────────────── Task Completion ────────────────────────────────────────────────╮\n",
-              "│                                                                                                                 │\n",
-              "│  Task Completed                                                                                                 │\n",
-              "│  Name: bd1a6f7d-9d37-47f0-98ce-2420c3175312                                                                     │\n",
-              "│  Agent: Sports Analyst                                                                                          │\n",
-              "│  Tool Args:                                                                                                     │\n",
-              "│                                                                                                                 │\n",
-              "│                                                                                                                 │\n",
-              "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\u001b[32m╭─\u001b[0m\u001b[32m───────────────────────────────────────────────\u001b[0m\u001b[32m Task Completion \u001b[0m\u001b[32m───────────────────────────────────────────────\u001b[0m\u001b[32m─╮\u001b[0m\n",
-              "\u001b[32m│\u001b[0m                                                                                                                 \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[1;32mTask Completed\u001b[0m                                                                                                 \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37mName: \u001b[0m\u001b[32mbd1a6f7d-9d37-47f0-98ce-2420c3175312\u001b[0m                                                                     \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37mAgent: \u001b[0m\u001b[32mSports Analyst\u001b[0m                                                                                          \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37mTool Args: \u001b[0m                                                                                                    \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m                                                                                                                 \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m                                                                                                                 \u001b[32m│\u001b[0m\n",
-              "\u001b[32m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n"
-            ],
-            "text/plain": []
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
╭────────────────────────────────────────────── 🧠 Retrieved Memory ──────────────────────────────────────────────╮\n",
-              "│                                                                                                                 │\n",
-              "│  Historical Data:                                                                                               │\n",
-              "│  - Ensure that the article includes direct quotes from Guardiola if possible to enhance credibility.            │\n",
-              "│  - Include more detailed statistical analysis or comparisons with previous seasons for a deeper insight into    │\n",
-              "│  the team's form.                                                                                               │\n",
-              "│  - Incorporate players' and experts' opinions or commentary to provide a well-rounded perspective.              │\n",
-              "│  - Add a section discussing future challenges or key upcoming matches for Manchester City.                      │\n",
-              "│  - Consider incorporating multimedia elements like images or videos ...                                         │\n",
-              "│                                                                                                                 │\n",
-              "╰─────────────────────────────────────────── Retrieval Time: 854.27ms ────────────────────────────────────────────╯\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\u001b[32m╭─\u001b[0m\u001b[32m─────────────────────────────────────────────\u001b[0m\u001b[32m 🧠 Retrieved Memory \u001b[0m\u001b[32m─────────────────────────────────────────────\u001b[0m\u001b[32m─╮\u001b[0m\n",
-              "\u001b[32m│\u001b[0m                                                                                                                 \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37mHistorical Data:\u001b[0m                                                                                               \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37m- Ensure that the article includes direct quotes from Guardiola if possible to enhance credibility.\u001b[0m            \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37m- Include more detailed statistical analysis or comparisons with previous seasons for a deeper insight into \u001b[0m   \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37mthe team's form.\u001b[0m                                                                                               \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37m- Incorporate players' and experts' opinions or commentary to provide a well-rounded perspective.\u001b[0m              \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37m- Add a section discussing future challenges or key upcoming matches for Manchester City.\u001b[0m                      \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37m- Consider incorporating multimedia elements like images or videos ...\u001b[0m                                         \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m                                                                                                                 \u001b[32m│\u001b[0m\n",
-              "\u001b[32m╰─\u001b[0m\u001b[32m──────────────────────────────────────────\u001b[0m\u001b[32m Retrieval Time: 854.27ms \u001b[0m\u001b[32m───────────────────────────────────────────\u001b[0m\u001b[32m─╯\u001b[0m\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
╭─────────────────────────────────────────────── 🤖 Agent Started ────────────────────────────────────────────────╮\n",
-              "│                                                                                                                 │\n",
-              "│  Agent: Sports Journalist                                                                                       │\n",
-              "│                                                                                                                 │\n",
-              "│  Task: Write a sports article about Manchester City's form using the analysis and Guardiola's comments.         │\n",
-              "│                                                                                                                 │\n",
-              "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\u001b[35m╭─\u001b[0m\u001b[35m──────────────────────────────────────────────\u001b[0m\u001b[35m 🤖 Agent Started \u001b[0m\u001b[35m───────────────────────────────────────────────\u001b[0m\u001b[35m─╮\u001b[0m\n",
-              "\u001b[35m│\u001b[0m                                                                                                                 \u001b[35m│\u001b[0m\n",
-              "\u001b[35m│\u001b[0m  \u001b[37mAgent: \u001b[0m\u001b[1;92mSports Journalist\u001b[0m                                                                                       \u001b[35m│\u001b[0m\n",
-              "\u001b[35m│\u001b[0m                                                                                                                 \u001b[35m│\u001b[0m\n",
-              "\u001b[35m│\u001b[0m  \u001b[37mTask: \u001b[0m\u001b[92mWrite a sports article about Manchester City's form using the analysis and Guardiola's comments.\u001b[0m         \u001b[35m│\u001b[0m\n",
-              "\u001b[35m│\u001b[0m                                                                                                                 \u001b[35m│\u001b[0m\n",
-              "\u001b[35m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n"
-            ],
-            "text/plain": []
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n"
-            ],
-            "text/plain": []
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
╭──────────────────────────────────────────────── Task Completion ────────────────────────────────────────────────╮\n",
-              "│                                                                                                                 │\n",
-              "│  Task Completed                                                                                                 │\n",
-              "│  Name: 8bcffe0e-5a64-4e12-8207-e0f8701d847b                                                                     │\n",
-              "│  Agent: Sports Journalist                                                                                       │\n",
-              "│  Tool Args:                                                                                                     │\n",
-              "│                                                                                                                 │\n",
-              "│                                                                                                                 │\n",
-              "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\u001b[32m╭─\u001b[0m\u001b[32m───────────────────────────────────────────────\u001b[0m\u001b[32m Task Completion \u001b[0m\u001b[32m───────────────────────────────────────────────\u001b[0m\u001b[32m─╮\u001b[0m\n",
-              "\u001b[32m│\u001b[0m                                                                                                                 \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[1;32mTask Completed\u001b[0m                                                                                                 \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37mName: \u001b[0m\u001b[32m8bcffe0e-5a64-4e12-8207-e0f8701d847b\u001b[0m                                                                     \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37mAgent: \u001b[0m\u001b[32mSports Journalist\u001b[0m                                                                                       \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m  \u001b[37mTool Args: \u001b[0m                                                                                                    \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m                                                                                                                 \u001b[32m│\u001b[0m\n",
-              "\u001b[32m│\u001b[0m                                                                                                                 \u001b[32m│\u001b[0m\n",
-              "\u001b[32m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "text/html": [
-              "
\n",
-              "
\n"
-            ],
-            "text/plain": [
-              "\n"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "\n",
-            "================================================================================\n",
-            "CREWAI AGENT MEMORY DEMO RESULT\n",
-            "================================================================================\n",
-            "**Manchester City’s Impeccable Form: A Reflection of Guardiola’s Philosophy**\n",
-            "\n",
-            "Manchester City has been turning heads with their exceptional form under the astute guidance of Pep Guardiola. The team’s recent performances have not only aligned seamlessly with their manager’s philosophy but have also placed them in a formidable position across various competitions. Guardiola himself expressed his satisfaction, stating, \"The team is playing well, we are in a good moment. The way we are training, the way we are playing - I am really pleased.\"\n",
-            "\n",
-            "City’s prowess has been evident both domestically and in international arenas. A key factor in their success is their meticulous training regimen, which has fostered strategic flexibility, a hallmark of Guardiola’s management. Over the past few matches, Manchester City has consistently maintained a high possession rate, often exceeding 60%. This high possession allows them to control the tempo and dictate the flow of the game, a crucial component of their strategy.\n",
-            "\n",
-            "A recent standout performance was their dominant victory against a top Premier League rival. In this match, City showcased their attacking capabilities and defensive solidity, managing to keep a clean sheet. The contributions of key players like Kevin De Bruyne and Erling Haaland have been instrumental. De Bruyne’s creativity and passing range have opened multiple avenues for attack, while Haaland’s clinical finishing has consistently troubled defenses.\n",
-            "\n",
-            "Guardiola’s system, which relies heavily on positional play and fluid movement, has been a critical factor in their ability to break down opposition defenses with quick, incisive passes. The team’s pressing game has also been a cornerstone of their strategy, allowing them to win back possession high up the pitch and quickly transition to attack.\n",
-            "\n",
-            "Despite the glowing form and Guardiola’s positive outlook, it’s important to acknowledge potential areas for improvement. While their attack is formidable, City has shown occasional vulnerability to counter-attacks, particularly when their full-backs are positioned high up the field. Addressing these defensive transitions will be crucial, especially against teams with quick counter-attacking capabilities.\n",
-            "\n",
-            "Looking forward, Manchester City’s current form is a strong foundation for upcoming challenges, including key fixtures in the Premier League and the knockout stages of the UEFA Champions League. Maintaining this performance level will be essential as they pursue multiple titles. The team’s depth, strategic versatility, and Guardiola’s leadership will be decisive factors in sustaining their momentum.\n",
-            "\n",
-            "In conclusion, Manchester City is indeed in a \"good moment,\" as Guardiola aptly puts it. Their recent performances reflect a well-oiled machine operating at high efficiency. However, the team must remain vigilant about potential weaknesses and continue adapting tactically to ensure their current form translates into long-term success. As they aim for glory, the synergy between Guardiola’s strategic mastermind and the players’ execution will undoubtedly be the key to their triumphs.\n",
-            "================================================================================\n",
-            "\n",
-            "✅ CrewAI agents completed successfully in 37.60 seconds!\n",
-            "✅ Agents used GSI-optimized Couchbase memory storage for fast retrieval!\n",
-            "✅ Memory will persist across sessions for continued learning and context retention!\n"
-          ]
-        }
-      ],
-      "source": [
-        "# Run the crew with optimized GSI memory\n",
-        "print(\"Running CrewAI agents with GSI-optimized memory storage...\")\n",
-        "start_time = time.time()\n",
-        "result = crew.kickoff()\n",
-        "execution_time = time.time() - start_time\n",
-        "\n",
-        "print(\"\\n\" + \"=\"*80)\n",
-        "print(\"CREWAI AGENT MEMORY DEMO RESULT\")\n",
-        "print(\"=\"*80)\n",
-        "print(result)\n",
-        "print(\"=\"*80)\n",
-        "print(f\"\\n✅ CrewAI agents completed successfully in {execution_time:.2f} seconds!\")\n",
-        "print(\"✅ Agents used GSI-optimized Couchbase memory storage for fast retrieval!\")\n",
-        "print(\"✅ Memory will persist across sessions for continued learning and context retention!\")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "id": "d4500466",
-      "metadata": {},
-      "source": [
-        "## Memory Retention Testing"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "id": "283e1d9e",
-      "metadata": {},
-      "source": [
-        "### Verify Memory Storage and Retrieval"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "id": "ed828a0f",
-      "metadata": {},
-      "source": [
-        "Test that our agents successfully stored memories and can retrieve them using semantic search."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 21,
-      "id": "558ac893",
-      "metadata": {},
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "\n",
-            "All memory entries in Couchbase:\n",
-            "--------------------------------------------------------------------------------\n",
-            "\n",
-            "Memory Search Results:\n",
-            "--------------------------------------------------------------------------------\n",
-            "Context: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a good moment. The way we are training, the way we are playing - I am really pleased.'\n",
-            "Distance: 0.285379886892123 (lower = more similar)\n",
-            "--------------------------------------------------------------------------------\n",
-            "Context: Manchester City's recent performance analysis under Pep Guardiola reflects a team in strong form and alignment with the manager's philosophy. Guardiola's comments, \"The team is playing well, we are in a good moment. The way we are training, the way we are playing - I am really pleased,\" suggest a high level of satisfaction with both the tactical execution and the overall team ethos on the pitch.\n",
-            "\n",
-            "In recent matches, Manchester City has demonstrated their prowess in both domestic and international competitions. This form can be attributed to their meticulous training regimen and strategic flexibility, hallmarks of Guardiola's management style. Over the past few matches, City has maintained a high possession rate, often exceeding 60%, which allows them to control the tempo and dictate the flow of the game. Their attacking prowess is underscored by their goal-scoring statistics, often leading the league in goals scored per match.\n",
-            "\n",
-            "One standout example of their performance is their recent dominant victory against a top Premier League rival, where they not only showcased their attacking capabilities but also their defensive solidity, keeping a clean sheet. Key players such as Kevin De Bruyne and Erling Haaland have been instrumental, with De Bruyne's creativity and passing range creating numerous opportunities, while Haaland's clinical finishing has consistently troubled defenses.\n",
-            "\n",
-            "Guardiola's system relies heavily on positional play and fluid movement, which has been evident in the team's ability to break down opposition defenses through quick, incisive passes. The team's pressing game has also been a critical component, often winning back possession high up the pitch and quickly transitioning to attack.\n",
-            "\n",
-            "Despite Guardiola's positive outlook, potential biases in his comments might overlook some areas needing improvement. For instance, while their attack is formidable, there have been instances where the team has shown vulnerability to counter-attacks, particularly when full-backs are pushed high up the field. Addressing these defensive transitions could be crucial, especially against teams with quick, counter-attacking capabilities.\n",
-            "\n",
-            "Looking ahead, Manchester City's current form sets a strong foundation for upcoming challenges, including key fixtures in the Premier League and the knockout stages of the UEFA Champions League. Maintaining this level of performance will be critical as they pursue multiple titles. The team's depth, strategic versatility, and Guardiola's leadership are likely to be decisive factors in sustaining their momentum.\n",
-            "\n",
-            "In summary, Manchester City is indeed in a \"good moment,\" as Guardiola states, with their recent performances reflecting a well-oiled machine operating at high efficiency. However, keeping a vigilant eye on potential weaknesses and continuing to adapt tactically will be essential to translating their current form into long-term success.\n",
-            "Distance: 0.22963345721993045 (lower = more similar)\n",
-            "--------------------------------------------------------------------------------\n",
-            "Context: **Manchester City’s Impeccable Form: A Reflection of Guardiola’s Philosophy**\n",
-            "\n",
-            "... (output truncated for brevity)\n"
-          ]
-        }
-      ],
-      "source": [
-        "# Wait for memories to be stored\n",
-        "time.sleep(2)\n",
-        "\n",
-        "# List all documents in the collection\n",
-        "try:\n",
-        "    # Query to fetch all documents of this memory type\n",
-        "    query_str = f\"SELECT META().id, * FROM `{storage.bucket_name}`.`{storage.scope_name}`.`{storage.collection_name}` WHERE memory_type = $type\"\n",
-        "    query_result = storage.cluster.query(query_str, type=storage.type)\n",
-        "    \n",
-        "    print(f\"\\nAll memory entries in Couchbase:\")\n",
-        "    print(\"-\" * 80)\n",
-        "    for i, row in enumerate(query_result, 1):\n",
-        "        doc_id = row.get('id')\n",
-        "        memory_id = row.get(storage.collection_name, {}).get('memory_id', 'unknown')\n",
-        "        content = row.get(storage.collection_name, {}).get('text', '')[:100] + \"...\"  # Truncate for readability\n",
-        "        \n",
-        "        print(f\"Entry {i}: {memory_id}\")\n",
-        "        print(f\"Content: {content}\")\n",
-        "        print(\"-\" * 80)\n",
-        "except Exception as e:\n",
-        "    print(f\"Failed to list memory entries: {str(e)}\")\n",
-        "\n",
-        "# Test memory retention\n",
-        "memory_query = \"What is Manchester City's current form according to Guardiola?\"\n",
-        "memory_results = storage.search(\n",
-        "    query=memory_query,\n",
-        "    limit=5,  # Increased to see more results\n",
-        "    score_threshold=0.0  # Lower threshold to see all results\n",
-        ")\n",
-        "\n",
-        "print(\"\\nMemory Search Results:\")\n",
-        "print(\"-\" * 80)\n",
-        "for result in memory_results:\n",
-        "    print(f\"Context: {result['context']}\")\n",
-        "    print(f\"Distance: {result['distance']} (lower = more similar)\")\n",
-        "    print(\"-\" * 80)\n",
-        "\n",
-        "# Try a more specific query to find agent interactions\n",
-        "interaction_query = \"Manchester City playing style analysis tactical\"\n",
-        "interaction_results = storage.search(\n",
-        "    query=interaction_query,\n",
-        "    limit=3,\n",
-        "    score_threshold=0.0\n",
-        ")\n",
-        "\n",
-        "print(\"\\nAgent Interaction Memory Results:\")\n",
-        "print(\"-\" * 80)\n",
-        "if interaction_results:\n",
-        "    for result in interaction_results:\n",
-        "        print(f\"Context: {result['context'][:200]}...\")  # Limit output size\n",
-        "        print(f\"Distance: {result['distance']} (lower = more similar)\")\n",
-        "        print(\"-\" * 80)\n",
-        "else:\n",
-        "    print(\"No interaction memories found. This is normal if agents haven't completed tasks yet.\")\n",
-        "    print(\"-\" * 80)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "id": "d23b2fbe",
-      "metadata": {},
-      "source": [
-        "## Conclusion"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "id": "d21915e5",
-      "metadata": {},
-      "source": [
-        "You've successfully implemented a custom memory backend for CrewAI agents using Couchbase GSI vector search!"
-      ]
-    }
-  ],
-  "metadata": {
-    "jupytext": {
-      "cell_metadata_filter": "-all",
-      "main_language": "python",
-      "notebook_metadata_filter": "-all"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 5
-}
diff --git a/crewai-short-term-memory/gsi/frontmatter.md b/crewai-short-term-memory/gsi/frontmatter.md
deleted file mode 100644
index 23cabbb7..00000000
--- a/crewai-short-term-memory/gsi/frontmatter.md
+++ /dev/null
@@ -1,22 +0,0 @@
----
-# frontmatter
-path: "/tutorial-crewai-short-term-memory-couchbase-with-global-secondary-index"
-title: Implementing Short-Term Memory for CrewAI Agents with Couchbase with GSI
-short_title: CrewAI Short-Term Memory with Couchbase with GSI
-description:
-  - Learn how to implement short-term memory for CrewAI agents using Couchbase's vector search capabilities with GSI.
-  - This tutorial demonstrates how to store and retrieve agent interactions using semantic search.
-  - You'll understand how to enhance CrewAI agents with memory capabilities using LangChain and Couchbase.
-content_type: tutorial
-filter: sdk
-technology:
-  - vector search
-tags:
-  - GSI
-  - Artificial Intelligence
-  - LangChain
-  - CrewAI
-sdk_language:
-  - python
-length: 45 Mins
----
diff --git a/crewai-short-term-memory/fts/.env.sample b/crewai-short-term-memory/query_based/.env.sample
similarity index 100%
rename from crewai-short-term-memory/fts/.env.sample
rename to crewai-short-term-memory/query_based/.env.sample
diff --git a/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb b/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb
new file mode 100644
index 00000000..9ace0da1
--- /dev/null
+++ b/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb
@@ -0,0 +1,1235 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "fa3af5ad",
+   "metadata": {},
+   "source": [
+    "## Introduction"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "32f885be",
+   "metadata": {},
+   "source": [
+    "## Prerequisites"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4cda6f3d",
+   "metadata": {},
+   "source": [
+    "### Couchbase Setup"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a3e10a74",
+   "metadata": {},
+   "source": [
+    "1. **Create Capella Account:** Deploy a [free tier cluster](https://cloud.couchbase.com/sign-up)\n",
+    "2. **Enable Query Service:** Required for Hyperscale and Composite Vector Indexes\n",
+    "3. **Configure Access:** Set up database credentials and network security\n",
+    "4. **Create Bucket:** Manual bucket creation recommended for Capella"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bb26dafe",
+   "metadata": {},
+   "source": [
+    "## Understanding Agent Memory"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "214ea40b",
+   "metadata": {},
+   "source": [
+    "### Why Memory Matters for AI Agents"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c3873132",
+   "metadata": {},
+   "source": [
+    "Memory in AI agents is a crucial capability that allows them to retain and utilize information across interactions, making them more effective and contextually aware. Without memory, agents would be limited to processing only the immediate input, lacking the ability to build upon past experiences or maintain continuity in conversations."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eaede747",
+   "metadata": {},
+   "source": [
+    "#### Types of Memory in AI Agents"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3346122d",
+   "metadata": {},
+   "source": [
+    "**Short-term Memory:**\n",
+    "- Retains recent interactions and context\n",
+    "- Typically spans the current conversation or session  \n",
+    "- Helps maintain coherence within a single interaction flow\n",
+    "- In CrewAI, this is what we're implementing with the Couchbase storage\n",
+    "\n",
+    "**Long-term Memory:**\n",
+    "- Stores persistent knowledge across multiple sessions\n",
+    "- Enables agents to recall past interactions even after long periods\n",
+    "- Helps build cumulative knowledge about users, preferences, and past decisions\n",
+    "- While this implementation is labeled as \"short-term memory\", the Couchbase storage backend can be effectively used for long-term memory as well, thanks to Couchbase's persistent storage capabilities and enterprise-grade durability features"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9d744f4a",
+   "metadata": {},
+   "source": [
+    "#### How Memory Works in Agents"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "53bd56b7",
+   "metadata": {},
+   "source": [
+    "Memory in AI agents typically involves:\n",
+    "- **Storage**: Information is encoded and stored in a database (like Couchbase, ChromaDB, or other vector stores)\n",
+    "- **Retrieval**: Relevant memories are fetched based on semantic similarity to current context\n",
+    "- **Integration**: Retrieved memories are incorporated into the agent's reasoning process\n",
+    "\n",
+    "The vector-based approach (using embeddings) is particularly powerful because it allows for semantic search - finding memories that are conceptually related to the current context, not just exact keyword matches."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "180704cb",
+   "metadata": {},
+   "source": [
+    "#### Benefits of Memory in AI Agents"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5242d1ea",
+   "metadata": {},
+   "source": [
+    "- **Contextual Understanding**: Agents can refer to previous parts of a conversation\n",
+    "- **Personalization**: Remembering user preferences and past interactions\n",
+    "- **Learning and Adaptation**: Building knowledge over time to improve responses\n",
+    "- **Task Continuity**: Resuming complex tasks across multiple interactions\n",
+    "- **Collaboration**: In multi-agent systems like CrewAI, memory enables agents to build on each other's work"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b6a39375",
+   "metadata": {},
+   "source": [
+    "#### Memory in CrewAI Specifically"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2f3f0133",
+   "metadata": {},
+   "source": [
+    "In CrewAI, memory serves several important functions:\n",
+    "- **Agent Specialization**: Each agent can maintain its own memory relevant to its expertise\n",
+    "- **Knowledge Transfer**: Agents can share insights through memory when collaborating on tasks\n",
+    "- **Process Continuity**: In sequential processes, later agents can access the work of earlier agents\n",
+    "- **Contextual Awareness**: Agents can reference previous findings when making decisions"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0082810e",
+   "metadata": {},
+   "source": [
+    "## Setup and Installation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c23683a4",
+   "metadata": {},
+   "source": [
+    "### Install Required Libraries"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b41c9376",
+   "metadata": {},
+   "source": [
+    "Install the necessary packages for CrewAI, Couchbase integration, and OpenAI embeddings."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "fd5f51cb",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m25.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.3\u001b[0m\n",
+      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
+      "Note: you may need to restart the kernel to use updated packages.\n"
+     ]
+    }
+   ],
+   "source": [
+    "%pip install --quiet crewai==0.186.1 langchain-couchbase==1.0.1 langchain-openai python-dotenv==1.1.1"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5e73ffeb",
+   "metadata": {},
+   "source": [
+    "### Import Required Modules"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bd67cca9",
+   "metadata": {},
+   "source": [
+    "Import libraries for CrewAI memory storage, Couchbase GSI vector search, and OpenAI embeddings."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "4fb688e4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from typing import Any, Dict, List, Optional\n",
+    "import os\n",
+    "import logging\n",
+    "from datetime import timedelta\n",
+    "from dotenv import load_dotenv\n",
+    "from crewai.memory.storage.rag_storage import RAGStorage\n",
+    "from crewai.memory.short_term.short_term_memory import ShortTermMemory\n",
+    "from crewai import Agent, Crew, Task, Process\n",
+    "from couchbase.cluster import Cluster\n",
+    "from couchbase.options import ClusterOptions\n",
+    "from couchbase.auth import PasswordAuthenticator\n",
+    "from couchbase.diagnostics import PingState, ServiceType\n",
+    "from langchain_couchbase.vectorstores import CouchbaseQueryVectorStore\n",
+    "from langchain_couchbase.vectorstores import DistanceStrategy\n",
+    "from langchain_couchbase.vectorstores import IndexType\n",
+    "from langchain_openai import OpenAIEmbeddings, ChatOpenAI\n",
+    "import time\n",
+    "import json\n",
+    "import uuid\n",
+    "\n",
+    "# Configure logging (disabled)\n",
+    "logging.basicConfig(level=logging.CRITICAL)\n",
+    "logger = logging.getLogger(__name__)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3c044af6",
+   "metadata": {},
+   "source": [
+    "### Environment Configuration"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "abe7a8ad",
+   "metadata": {},
+   "source": [
+    "Configure environment variables for secure access to Couchbase and OpenAI services. Create a `.env` file with your credentials."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "a1d82bff",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "load_dotenv(\"./.env\")\n",
+    "\n",
+    "# Verify environment variables\n",
+    "required_vars = ['OPENAI_API_KEY', 'CB_HOST', 'CB_USERNAME', 'CB_PASSWORD']\n",
+    "for var in required_vars:\n",
+    "    if not os.getenv(var):\n",
+    "        raise ValueError(f\"{var} environment variable is required\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a6c46413",
+   "metadata": {},
+   "source": [
+    "## Understanding Hyperscale and Composite Vector Indexes"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2ff0e7b8",
+   "metadata": {},
+   "source": [
+    "### Vector Index Types"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fde149b9",
+   "metadata": {},
+   "source": [
+    "Couchbase offers two types of vector indexes for different use cases:\n",
+    "\n",
+    "**Hyperscale Vector Indexes:**\n",
+    "- Best for pure vector searches - content discovery, recommendations, semantic search\n",
+    "- High performance with low memory footprint - designed to scale to billions of vectors\n",
+    "- Optimized for concurrent operations - supports simultaneous searches and inserts\n",
+    "- Use when: You primarily perform vector-only queries without complex scalar filtering\n",
+    "- Ideal for: Large-scale semantic search, recommendation systems, content discovery\n",
+    "\n",
+    "**Composite Vector Indexes:**\n",
+    "- Best for filtered vector searches - combines vector search with scalar value filtering\n",
+    "- Efficient pre-filtering - scalar attributes reduce the vector comparison scope\n",
+    "- Use when: Your queries combine vector similarity with scalar filters that eliminate large portions of data\n",
+    "- Ideal for: Compliance-based filtering, user-specific searches, time-bounded queries\n",
+    "\n",
+    "For this CrewAI memory implementation, we'll use **Hyperscale Vector Index** as it's optimized for pure semantic search scenarios typical in AI agent memory systems."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1dfc28e9",
+   "metadata": {},
+   "source": [
+    "### Understanding Index Configuration"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "acab2b26",
+   "metadata": {},
+   "source": [
+    "The `index_description` parameter controls how Couchbase optimizes vector storage and search performance through centroids and quantization:\n",
+    "\n",
+    "**Format**: `'IVF[],{PQ|SQ}'`\n",
+    "\n",
+    "**Centroids (IVF - Inverted File):**\n",
+    "- Controls how the dataset is subdivided for faster searches\n",
+    "- More centroids = faster search, slower training  \n",
+    "- Fewer centroids = slower search, faster training\n",
+    "- If omitted (like IVF,SQ8), Couchbase auto-selects based on dataset size\n",
+    "\n",
+    "**Quantization Options:**\n",
+    "- SQ (Scalar Quantization): SQ4, SQ6, SQ8 (4, 6, or 8 bits per dimension)\n",
+    "- PQ (Product Quantization): PQx (e.g., PQ32x8)\n",
+    "- Higher values = better accuracy, larger index size\n",
+    "\n",
+    "**Common Examples:**\n",
+    "- IVF,SQ8 - Auto centroids, 8-bit scalar quantization (good default)\n",
+    "- IVF1000,SQ6 - 1000 centroids, 6-bit scalar quantization  \n",
+    "- IVF,PQ32x8 - Auto centroids, 32 subquantizers with 8 bits\n",
+    "\n",
+    "For detailed configuration options, see the [Quantization & Centroid Settings](https://docs.couchbase.com/cloud/vector-index/hyperscale-vector-index.html#algo_settings).\n",
+    "\n",
+    "For more information on Hyperscale and Composite Vector Indexes, see the [Couchbase Vector Index Documentation](https://docs.couchbase.com/cloud/vector-index/use-vector-indexes.html).\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3c7f0633",
+   "metadata": {},
+   "source": [
+    "## CouchbaseStorage Implementation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5df3792c",
+   "metadata": {},
+   "source": [
+    "### CouchbaseStorage Class"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a5e8abec",
+   "metadata": {},
+   "source": [
+    "This class extends CrewAI's `RAGStorage` to provide Hyperscale and Composite Vector Index search capabilities for agent memory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b29c4840",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class CouchbaseStorage(RAGStorage):\n",
+    "    \"\"\"\n",
+    "    Extends RAGStorage to handle embeddings for memory entries using Couchbase Hyperscale and Composite Vector Indexes.\n",
+    "    \"\"\"\n",
+    "\n",
+    "    def __init__(self, type: str, allow_reset: bool = True, embedder_config: Optional[Dict[str, Any]] = None, crew: Optional[Any] = None):\n",
+    "        \"\"\"Initialize CouchbaseStorage with vector index search configuration.\"\"\"\n",
+    "        super().__init__(type, allow_reset, embedder_config, crew)\n",
+    "        self._initialize_app()\n",
+    "\n",
+    "    def search(\n",
+    "        self,\n",
+    "        query: str,\n",
+    "        limit: int = 3,\n",
+    "        filter: Optional[dict] = None,\n",
+    "        score_threshold: float = 0,\n",
+    "    ) -> List[Dict[str, Any]]:\n",
+    "        \"\"\"\n",
+    "        Search memory entries using vector similarity.\n",
+    "        \"\"\"\n",
+    "        try:\n",
+    "            # Add type filter\n",
+    "            search_filter = {\"memory_type\": self.type}\n",
+    "            if filter:\n",
+    "                search_filter.update(filter)\n",
+    "\n",
+    "            # Execute search using vector index\n",
+    "            results = self.vector_store.similarity_search_with_score(\n",
+    "                query,\n",
+    "                k=limit,\n",
+    "                filter=search_filter\n",
+    "            )\n",
+    "            \n",
+    "            # Format results and deduplicate by content\n",
+    "            seen_contents = set()\n",
+    "            formatted_results = []\n",
+    "            \n",
+    "            for i, (doc, distance) in enumerate(results):\n",
+    "                # Note: Lower distance indicates higher similarity\n",
+    "                if distance <= (1.0 - score_threshold):  # Convert threshold for distance metric\n",
+    "                    content = doc.page_content\n",
+    "                    if content not in seen_contents:\n",
+    "                        seen_contents.add(content)\n",
+    "                        formatted_results.append({\n",
+    "                            \"id\": doc.metadata.get(\"memory_id\", str(i)),\n",
+    "                            \"metadata\": doc.metadata,\n",
+    "                            \"context\": content,\n",
+    "                            \"distance\": float(distance)\n",
+    "                        })\n",
+    "            \n",
+    "            logger.info(f\"Found {len(formatted_results)} unique results for query: {query}\")\n",
+    "            return formatted_results\n",
+    "\n",
+    "        except Exception as e:\n",
+    "            logger.error(f\"Search failed: {str(e)}\")\n",
+    "            return []\n",
+    "\n",
+    "    def save(self, value: Any, metadata: Dict[str, Any]) -> None:\n",
+    "        \"\"\"\n",
+    "        Save a memory entry with metadata.\n",
+    "        \"\"\"\n",
+    "        try:\n",
+    "            # Generate unique ID\n",
+    "            memory_id = str(uuid.uuid4())\n",
+    "            timestamp = int(time.time() * 1000)\n",
+    "            \n",
+    "            # Prepare metadata (create a copy to avoid modifying references)\n",
+    "            if not metadata:\n",
+    "                metadata = {}\n",
+    "            else:\n",
+    "                metadata = metadata.copy()  # Create a copy to avoid modifying references\n",
+    "                \n",
+    "            # Process agent-specific information if present\n",
+    "            agent_name = metadata.get('agent', 'unknown')\n",
+    "                \n",
+    "            # Clean up value if it has the typical LLM response format\n",
+    "            value_str = str(value)\n",
+    "            if \"Final Answer:\" in value_str:\n",
+    "                # Extract just the actual content - everything after \"Final Answer:\"\n",
+    "                parts = value_str.split(\"Final Answer:\", 1)\n",
+    "                if len(parts) > 1:\n",
+    "                    value = parts[1].strip()\n",
+    "                    logger.info(f\"Cleaned up response format for agent: {agent_name}\")\n",
+    "            elif value_str.startswith(\"Thought:\"):\n",
+    "                # Handle thought/final answer format\n",
+    "                if \"Final Answer:\" in value_str:\n",
+    "                    parts = value_str.split(\"Final Answer:\", 1)\n",
+    "                    if len(parts) > 1:\n",
+    "                        value = parts[1].strip()\n",
+    "                        logger.info(f\"Cleaned up thought process format for agent: {agent_name}\")\n",
+    "            \n",
+    "            # Update metadata\n",
+    "            metadata.update({\n",
+    "                \"memory_id\": memory_id,\n",
+    "                \"memory_type\": self.type,\n",
+    "                \"timestamp\": timestamp,\n",
+    "                \"source\": \"crewai\"\n",
+    "            })\n",
+    "\n",
+    "            # Log memory information for debugging\n",
+    "            value_preview = str(value)[:100] + \"...\" if len(str(value)) > 100 else str(value)\n",
+    "            metadata_preview = {k: v for k, v in metadata.items() if k != \"embedding\"}\n",
+    "            logger.info(f\"Saving memory for Agent: {agent_name}\")\n",
+    "            logger.info(f\"Memory value preview: {value_preview}\")\n",
+    "            logger.info(f\"Memory metadata: {metadata_preview}\")\n",
+    "            \n",
+    "            # Convert value to string if needed\n",
+    "            if isinstance(value, (dict, list)):\n",
+    "                value = json.dumps(value)\n",
+    "            elif not isinstance(value, str):\n",
+    "                value = str(value)\n",
+    "\n",
+    "            # Save to vector store\n",
+    "            self.vector_store.add_texts(\n",
+    "                texts=[value],\n",
+    "                metadatas=[metadata],\n",
+    "                ids=[memory_id]\n",
+    "            )\n",
+    "            logger.info(f\"Saved memory {memory_id}: {value[:100]}...\")\n",
+    "\n",
+    "        except Exception as e:\n",
+    "            logger.error(f\"Save failed: {str(e)}\")\n",
+    "            raise\n",
+    "\n",
+    "    def reset(self) -> None:\n",
+    "        \"\"\"Reset the memory storage if allowed.\"\"\"\n",
+    "        if not self.allow_reset:\n",
+    "            return\n",
+    "\n",
+    "        try:\n",
+    "            # Delete documents of this memory type\n",
+    "            self.cluster.query(\n",
+    "                f\"DELETE FROM `{self.bucket_name}`.`{self.scope_name}`.`{self.collection_name}` WHERE memory_type = $type\",\n",
+    "                type=self.type\n",
+    "            ).execute()\n",
+    "            logger.info(f\"Reset memory type: {self.type}\")\n",
+    "        except Exception as e:\n",
+    "            logger.error(f\"Reset failed: {str(e)}\")\n",
+    "            raise\n",
+    "\n",
+    "    def _initialize_app(self):\n",
+    "        \"\"\"Initialize Couchbase connection and vector store.\"\"\"\n",
+    "        try:\n",
+    "            # Initialize embeddings\n",
+    "            if self.embedder_config and self.embedder_config.get(\"provider\") == \"openai\":\n",
+    "                self.embeddings = OpenAIEmbeddings(\n",
+    "                    openai_api_key=os.getenv('OPENAI_API_KEY'),\n",
+    "                    model=self.embedder_config.get(\"config\", {}).get(\"model\", \"text-embedding-3-small\")\n",
+    "                )\n",
+    "            else:\n",
+    "                self.embeddings = OpenAIEmbeddings(\n",
+    "                    openai_api_key=os.getenv('OPENAI_API_KEY'),\n",
+    "                    model=\"text-embedding-3-small\"\n",
+    "                )\n",
+    "\n",
+    "            # Connect to Couchbase\n",
+    "            auth = PasswordAuthenticator(\n",
+    "                os.getenv('CB_USERNAME', ''),\n",
+    "                os.getenv('CB_PASSWORD', '')\n",
+    "            )\n",
+    "            options = ClusterOptions(auth)\n",
+    "            \n",
+    "            # Initialize cluster connection\n",
+    "            self.cluster = Cluster(os.getenv('CB_HOST', ''), options)\n",
+    "            self.cluster.wait_until_ready(timedelta(seconds=5))\n",
+    "\n",
+    "            # Check Query service (required for Hyperscale and Composite Vector Indexes)\n",
+    "            ping_result = self.cluster.ping()\n",
+    "            query_available = False\n",
+    "            for service_type, endpoints in ping_result.endpoints.items():\n",
+    "                if service_type.name == 'Query':  # Query Service for Hyperscale and Composite Vector Indexes\n",
+    "                    for endpoint in endpoints:\n",
+    "                        if endpoint.state == PingState.OK:\n",
+    "                            query_available = True\n",
+    "                            logger.info(f\"Query service is responding at: {endpoint.remote}\")\n",
+    "                            break\n",
+    "                    break\n",
+    "            if not query_available:\n",
+    "                raise RuntimeError(\"Query service not found or not responding. Hyperscale and Composite Vector Indexes require Query Service.\")\n",
+    "            \n",
+    "            # Set up storage configuration\n",
+    "            self.bucket_name = os.getenv('CB_BUCKET_NAME', 'vector-search-testing')\n",
+    "            self.scope_name = os.getenv('CB_SCOPE_NAME', 'shared')\n",
+    "            self.collection_name = os.getenv('COLLECTION_NAME', 'crew')\n",
+    "            self.index_name = os.getenv('INDEX_NAME', 'vector_search_crew_gsi')\n",
+    "\n",
+    "            # Initialize vector store\n",
+    "            self.vector_store = CouchbaseQueryVectorStore(\n",
+    "                cluster=self.cluster,\n",
+    "                bucket_name=self.bucket_name,\n",
+    "                scope_name=self.scope_name,\n",
+    "                collection_name=self.collection_name,\n",
+    "                embedding=self.embeddings,\n",
+    "                distance_metric=DistanceStrategy.COSINE,\n",
+    "            )\n",
+    "            logger.info(f\"Initialized CouchbaseStorage for type: {self.type}\")\n",
+    "\n",
+    "        except Exception as e:\n",
+    "            logger.error(f\"Initialization failed: {str(e)}\")\n",
+    "            raise"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3566d5bf",
+   "metadata": {},
+   "source": [
+    "## Memory Search Performance Testing"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ff154822",
+   "metadata": {},
+   "source": [
+    "Now let's demonstrate the performance benefits of vector index optimization by testing pure memory search performance. We'll compare three optimization levels:\n",
+    "\n",
+    "1. **Baseline Performance**: Memory search without vector index optimization\n",
+    "2. **Optimized Performance**: Same search with Hyperscale Vector Index\n",
+    "3. **Cache Benefits**: Show how caching can be applied on top of vector indexes for repeated queries\n",
+    "\n",
+    "**Important**: This testing focuses on pure memory search performance, isolating the vector index improvements from CrewAI agent workflow overhead."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "29717ea7",
+   "metadata": {},
+   "source": [
+    "### Initialize Storage and Test Functions"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7f41c284",
+   "metadata": {},
+   "source": [
+    "First, let's set up the storage and create test functions for measuring memory search performance."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "06349452",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initialize storage\n",
+    "storage = CouchbaseStorage(\n",
+    "    type=\"short_term\",\n",
+    "    embedder_config={\n",
+    "        \"provider\": \"openai\",\n",
+    "        \"config\": {\"model\": \"text-embedding-3-small\"}\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "# Reset storage\n",
+    "storage.reset()\n",
+    "\n",
+    "# Test storage\n",
+    "test_memory = \"Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a good moment. The way we are training, the way we are playing - I am really pleased.'\"\n",
+    "test_metadata = {\"category\": \"sports\", \"test\": \"initial_memory\"}\n",
+    "storage.save(test_memory, test_metadata)\n",
+    "\n",
+    "import time\n",
+    "\n",
+    "def test_memory_search_performance(storage, query, label=\"Memory Search\"):\n",
+    "    \"\"\"Test pure memory search performance and return timing metrics\"\"\"\n",
+    "    print(f\"\\n[{label}] Testing memory search performance\")\n",
+    "    print(f\"[{label}] Query: '{query}'\")\n",
+    "    \n",
+    "    start_time = time.time()\n",
+    "    \n",
+    "    try:\n",
+    "        results = storage.search(query, limit=3)\n",
+    "        end_time = time.time()\n",
+    "        search_time = end_time - start_time\n",
+    "        \n",
+    "        print(f\"[{label}] Memory search completed in {search_time:.4f} seconds\")\n",
+    "        print(f\"[{label}] Found {len(results)} memories\")\n",
+    "        \n",
+    "        if results:\n",
+    "            print(f\"[{label}] Top result distance: {results[0]['distance']:.6f} (lower = more similar)\")\n",
+    "            preview = results[0]['context'][:100] + \"...\" if len(results[0]['context']) > 100 else results[0]['context']\n",
+    "            print(f\"[{label}] Top result preview: {preview}\")\n",
+    "        \n",
+    "        return search_time\n",
+    "    except Exception as e:\n",
+    "        print(f\"[{label}] Memory search failed: {str(e)}\")\n",
+    "        return None"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "198a7939",
+   "metadata": {},
+   "source": [
+    "### Test 1: Baseline Performance (No Vector Index)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ef5d4fde",
+   "metadata": {},
+   "source": [
+    "Test pure memory search performance without vector index optimization."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "383bb87d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test baseline memory search performance without vector index\n",
+    "test_query = \"What did Guardiola say about Manchester City?\"\n",
+    "print(\"Testing baseline memory search performance without vector index optimization...\")\n",
+    "baseline_time = test_memory_search_performance(storage, test_query, \"Baseline Search\")\n",
+    "print(f\"\\nBaseline memory search time (without vector index): {baseline_time:.4f} seconds\\n\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a88e1719",
+   "metadata": {},
+   "source": [
+    "### Create Hyperscale Vector Index"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "be7acf07",
+   "metadata": {},
+   "source": [
+    "Now let's create a Hyperscale Vector Index to enable high-performance memory searches. The index creation is done programmatically through the vector store."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bde97a46",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create Hyperscale Vector Index for optimal performance\n",
+    "print(\"Creating Hyperscale Vector Index...\")\n",
+    "try:\n",
+    "    storage.vector_store.create_index(\n",
+    "        index_type=IndexType.HYPERSCALE,\n",
+    "        # index_type=IndexType.COMPOSITE,  # Uncomment this line to create a Composite Vector Index instead\n",
+    "        index_name=storage.index_name,\n",
+    "        index_description=\"IVF,SQ8\"  # Auto-selected centroids with 8-bit scalar quantization\n",
+    "    )\n",
+    "    print(f\"Hyperscale Vector Index created successfully: {storage.index_name}\")\n",
+    "    \n",
+    "    # Wait for index to become available\n",
+    "    print(\"Waiting for index to become available...\")\n",
+    "    time.sleep(5)\n",
+    "    \n",
+    "except Exception as e:\n",
+    "    if \"already exists\" in str(e).lower():\n",
+    "        print(f\"Vector index '{storage.index_name}' already exists, proceeding...\")\n",
+    "    else:\n",
+    "        print(f\"Error creating vector index: {str(e)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c389eecb",
+   "metadata": {},
+   "source": [
+    "### Alternative: Composite Index Configuration"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4e7555da",
+   "metadata": {},
+   "source": [
+    "If your agent memory use case requires complex filtering with scalar attributes, you can create a **Composite Vector Index** instead by changing the configuration above:\n",
+    "\n",
+    "```python\n",
+    "# Alternative: Create a Composite Vector Index for filtered memory searches\n",
+    "storage.vector_store.create_index(\n",
+    "    index_type=IndexType.COMPOSITE,  # Instead of IndexType.BHIVE\n",
+    "    index_name=storage.index_name,\n",
+    "    index_description=\"IVF,SQ8\"      # Same quantization settings\n",
+    ")\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8e719352",
+   "metadata": {},
+   "source": [
+    "### Test 2: Vector Index-Optimized Performance"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5d786f04",
+   "metadata": {},
+   "source": [
+    "Test the same memory search with Hyperscale Vector Index (BHIVE) optimization."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "849758ae",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test memory search performance with Hyperscale Vector Index\n",
+    "print(\"Testing memory search performance with Hyperscale Vector Index optimization...\")\n",
+    "optimized_time = test_memory_search_performance(storage, test_query, \"Vector Index-Optimized Search\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "905cf62e",
+   "metadata": {},
+   "source": [
+    "### Test 3: Cache Benefits Testing"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a704c5c1",
+   "metadata": {},
+   "source": [
+    "Now let's demonstrate how caching can improve performance for repeated queries. **Note**: Caching benefits apply to both baseline and vector index-optimized searches."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "febeab1f",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Testing cache benefits with memory search...\n",
+      "First execution (cache miss):\n",
+      "\n",
+      "[Cache Test - First Run] Testing memory search performance\n",
+      "[Cache Test - First Run] Query: 'How is Manchester City performing in training sessions?'\n",
+      "[Cache Test - First Run] Memory search completed in 0.6076 seconds\n",
+      "[Cache Test - First Run] Found 1 memories\n",
+      "[Cache Test - First Run] Top result distance: 0.379242 (lower = more similar)\n",
+      "[Cache Test - First Run] Top result preview: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a ...\n",
+      "\n",
+      "Second execution (cache hit - should be faster):\n",
+      "\n",
+      "[Cache Test - Second Run] Testing memory search performance\n",
+      "[Cache Test - Second Run] Query: 'How is Manchester City performing in training sessions?'\n",
+      "[Cache Test - Second Run] Memory search completed in 0.4745 seconds\n",
+      "[Cache Test - Second Run] Found 1 memories\n",
+      "[Cache Test - Second Run] Top result distance: 0.379200 (lower = more similar)\n",
+      "[Cache Test - Second Run] Top result preview: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a ...\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Test cache benefits with a different query to avoid interference\n",
+    "cache_test_query = \"How is Manchester City performing in training sessions?\"\n",
+    "\n",
+    "print(\"Testing cache benefits with memory search...\")\n",
+    "print(\"First execution (cache miss):\")\n",
+    "cache_time_1 = test_memory_search_performance(storage, cache_test_query, \"Cache Test - First Run\")\n",
+    "\n",
+    "print(\"\\nSecond execution (cache hit - should be faster):\")\n",
+    "cache_time_2 = test_memory_search_performance(storage, cache_test_query, \"Cache Test - Second Run\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0cd9de44",
+   "metadata": {},
+   "source": [
+    "### Memory Search Performance Analysis"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f475ccc3",
+   "metadata": {},
+   "source": [
+    "Let's analyze the memory search performance improvements across all optimization levels:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f813eb1a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(\"\\n\" + \"=\"*80)\n",
+    "print(\"MEMORY SEARCH PERFORMANCE OPTIMIZATION SUMMARY\")\n",
+    "print(\"=\"*80)\n",
+    "\n",
+    "print(f\"Phase 1 - Baseline Search (No Vector Index):     {baseline_time:.4f} seconds\")\n",
+    "print(f\"Phase 2 - Vector Index-Optimized Search:         {optimized_time:.4f} seconds\")\n",
+    "if cache_time_1 and cache_time_2:\n",
+    "    print(f\"Phase 3 - Cache Benefits:\")\n",
+    "    print(f\"  First execution (cache miss):         {cache_time_1:.4f} seconds\")\n",
+    "    print(f\"  Second execution (cache hit):         {cache_time_2:.4f} seconds\")\n",
+    "\n",
+    "print(\"\\n\" + \"-\"*80)\n",
+    "print(\"MEMORY SEARCH OPTIMIZATION IMPACT:\")\n",
+    "print(\"-\"*80)\n",
+    "\n",
+    "# Vector index improvement analysis\n",
+    "if baseline_time and optimized_time:\n",
+    "    speedup = baseline_time / optimized_time if optimized_time > 0 else float('inf')\n",
+    "    time_saved = baseline_time - optimized_time\n",
+    "    percent_improvement = (time_saved / baseline_time) * 100\n",
+    "    print(f\"Vector Index Benefit:      {speedup:.2f}x faster ({percent_improvement:.1f}% improvement)\")\n",
+    "\n",
+    "# Cache improvement analysis\n",
+    "if cache_time_1 and cache_time_2 and cache_time_2 < cache_time_1:\n",
+    "    cache_speedup = cache_time_1 / cache_time_2\n",
+    "    cache_improvement = ((cache_time_1 - cache_time_2) / cache_time_1) * 100\n",
+    "    print(f\"Cache Benefit:          {cache_speedup:.2f}x faster ({cache_improvement:.1f}% improvement)\")\n",
+    "else:\n",
+    "    print(f\"Cache Benefit:          Variable (depends on query complexity and caching mechanism)\")\n",
+    "\n",
+    "print(f\"\\nKey Insights for Agent Memory Performance:\")\n",
+    "print(f\"• Hyperscale Vector Indexes provide significant performance improvements for memory search\")\n",
+    "print(f\"• Performance gains are most dramatic for complex semantic memory queries\")\n",
+    "print(f\"• Hyperscale optimization is particularly effective for agent conversational memory\")\n",
+    "print(f\"• Combined with proper quantization (SQ8), vector indexes deliver production-ready performance\")\n",
+    "print(f\"• These performance improvements directly benefit agent response times and scalability\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c4b069f8",
+   "metadata": {},
+   "source": [
+    "**Note on Hyperscale Vector Index Performance:** The Hyperscale Vector Index (BHIVE) may show slower performance for very small datasets (few documents) due to the additional overhead of maintaining the index structure. However, as the dataset scales up, the Hyperscale Vector Index becomes significantly faster than traditional vector searches. The initial overhead investment pays off dramatically with larger memory stores, making it essential for production agent deployments with substantial conversational history."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "126d4fcf",
+   "metadata": {},
+   "source": [
+    "## CrewAI Agent Memory Demo"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a3c67329",
+   "metadata": {},
+   "source": [
+    "### What is CrewAI Agent Memory?"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8f71f9ec",
+   "metadata": {},
+   "source": [
+    "Now that we've optimized our memory search performance, let's demonstrate how CrewAI agents can leverage this vector index-optimized memory system. CrewAI agent memory enables:\n",
+    "\n",
+    "- **Persistent Context**: Agents remember information across conversations and tasks\n",
+    "- **Semantic Recall**: Agents can find relevant memories using natural language queries\n",
+    "- **Collaborative Memory**: Multiple agents can share and build upon each other's memories\n",
+    "- **Performance Benefits**: Our vector index optimizations directly improve agent memory retrieval speed\n",
+    "\n",
+    "This demo shows how the memory performance improvements we validated translate to real agent workflows."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0ea8887d",
+   "metadata": {},
+   "source": [
+    "### Create Agents with Optimized Memory"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bdf480e7",
+   "metadata": {},
+   "source": [
+    "Set up CrewAI agents that use our vector index-optimized Couchbase memory storage for fast, contextual memory retrieval."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "509767fb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initialize ShortTermMemory with our storage\n",
+    "memory = ShortTermMemory(storage=storage)\n",
+    "\n",
+    "# Initialize language model\n",
+    "llm = ChatOpenAI(\n",
+    "    model=\"gpt-4o\",\n",
+    "    temperature=0.7\n",
+    ")\n",
+    "\n",
+    "# Create agents with memory\n",
+    "sports_analyst = Agent(\n",
+    "    role='Sports Analyst',\n",
+    "    goal='Analyze Manchester City performance',\n",
+    "    backstory='Expert at analyzing football teams and providing insights on their performance',\n",
+    "    llm=llm,\n",
+    "    memory=True,\n",
+    "    memory_storage=memory\n",
+    ")\n",
+    "\n",
+    "journalist = Agent(\n",
+    "    role='Sports Journalist',\n",
+    "    goal='Create engaging football articles',\n",
+    "    backstory='Experienced sports journalist who specializes in Premier League coverage',\n",
+    "    llm=llm,\n",
+    "    memory=True,\n",
+    "    memory_storage=memory\n",
+    ")\n",
+    "\n",
+    "# Create tasks\n",
+    "analysis_task = Task(\n",
+    "    description='Analyze Manchester City\\'s recent performance based on Pep Guardiola\\'s comments: \"The team is playing well, we are in a good moment. The way we are training, the way we are playing - I am really pleased.\"',\n",
+    "    agent=sports_analyst,\n",
+    "    expected_output=\"A comprehensive analysis of Manchester City's current form based on Guardiola's comments.\"\n",
+    ")\n",
+    "\n",
+    "writing_task = Task(\n",
+    "    description='Write a sports article about Manchester City\\'s form using the analysis and Guardiola\\'s comments.',\n",
+    "    agent=journalist,\n",
+    "    context=[analysis_task],\n",
+    "    expected_output=\"An engaging sports article about Manchester City's current form and Guardiola's perspective.\"\n",
+    ")\n",
+    "\n",
+    "# Create crew with memory\n",
+    "crew = Crew(\n",
+    "    agents=[sports_analyst, journalist],\n",
+    "    tasks=[analysis_task, writing_task],\n",
+    "    process=Process.sequential,\n",
+    "    memory=True,\n",
+    "    short_term_memory=memory,  # Explicitly pass our memory implementation\n",
+    "    verbose=True\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "950636f7",
+   "metadata": {},
+   "source": [
+    "### Run Agent Memory Demo"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "95c612da",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Run the crew with optimized vector index memory\n",
+    "print(\"Running CrewAI agents with vector index-optimized memory storage...\")\n",
+    "start_time = time.time()\n",
+    "result = crew.kickoff()\n",
+    "execution_time = time.time() - start_time\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*80)\n",
+    "print(\"CREWAI AGENT MEMORY DEMO RESULT\")\n",
+    "print(\"=\"*80)\n",
+    "print(result)\n",
+    "print(\"=\"*80)\n",
+    "print(f\"\\n✅ CrewAI agents completed successfully in {execution_time:.2f} seconds!\")\n",
+    "print(\"✅ Agents used vector index-optimized Couchbase memory storage for fast retrieval!\")\n",
+    "print(\"✅ Memory will persist across sessions for continued learning and context retention!\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d4500466",
+   "metadata": {},
+   "source": [
+    "## Memory Retention Testing"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "283e1d9e",
+   "metadata": {},
+   "source": [
+    "### Verify Memory Storage and Retrieval"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ed828a0f",
+   "metadata": {},
+   "source": [
+    "Test that our agents successfully stored memories and can retrieve them using semantic search."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "558ac893",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "All memory entries in Couchbase:\n",
+      "--------------------------------------------------------------------------------\n",
+      "\n",
+      "Memory Search Results:\n",
+      "--------------------------------------------------------------------------------\n",
+      "Context: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a good moment. The way we are training, the way we are playing - I am really pleased.'\n",
+      "Distance: 0.285379886892123 (lower = more similar)\n",
+      "--------------------------------------------------------------------------------\n",
+      "Context: Manchester City's recent performance analysis under Pep Guardiola reflects a team in strong form and alignment with the manager's philosophy. Guardiola's comments, \"The team is playing well, we are in a good moment. The way we are training, the way we are playing - I am really pleased,\" suggest a high level of satisfaction with both the tactical execution and the overall team ethos on the pitch.\n",
+      "\n",
+      "In recent matches, Manchester City has demonstrated their prowess in both domestic and international competitions. This form can be attributed to their meticulous training regimen and strategic flexibility, hallmarks of Guardiola's management style. Over the past few matches, City has maintained a high possession rate, often exceeding 60%, which allows them to control the tempo and dictate the flow of the game. Their attacking prowess is underscored by their goal-scoring statistics, often leading the league in goals scored per match.\n",
+      "\n",
+      "One standout example of their performance is their recent dominant victory against a top Premier League rival, where they not only showcased their attacking capabilities but also their defensive solidity, keeping a clean sheet. Key players such as Kevin De Bruyne and Erling Haaland have been instrumental, with De Bruyne's creativity and passing range creating numerous opportunities, while Haaland's clinical finishing has consistently troubled defenses.\n",
+      "\n",
+      "Guardiola's system relies heavily on positional play and fluid movement, which has been evident in the team's ability to break down opposition defenses through quick, incisive passes. The team's pressing game has also been a critical component, often winning back possession high up the pitch and quickly transitioning to attack.\n",
+      "\n",
+      "Despite Guardiola's positive outlook, potential biases in his comments might overlook some areas needing improvement. For instance, while their attack is formidable, there have been instances where the team has shown vulnerability to counter-attacks, particularly when full-backs are pushed high up the field. Addressing these defensive transitions could be crucial, especially against teams with quick, counter-attacking capabilities.\n",
+      "\n",
+      "Looking ahead, Manchester City's current form sets a strong foundation for upcoming challenges, including key fixtures in the Premier League and the knockout stages of the UEFA Champions League. Maintaining this level of performance will be critical as they pursue multiple titles. The team's depth, strategic versatility, and Guardiola's leadership are likely to be decisive factors in sustaining their momentum.\n",
+      "\n",
+      "In summary, Manchester City is indeed in a \"good moment,\" as Guardiola states, with their recent performances reflecting a well-oiled machine operating at high efficiency. However, keeping a vigilant eye on potential weaknesses and continuing to adapt tactically will be essential to translating their current form into long-term success.\n",
+      "Distance: 0.22963345721993045 (lower = more similar)\n",
+      "--------------------------------------------------------------------------------\n",
+      "Context: **Manchester City’s Impeccable Form: A Reflection of Guardiola’s Philosophy**\n",
+      "\n",
+      "... (output truncated for brevity)\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Wait for memories to be stored\n",
+    "time.sleep(2)\n",
+    "\n",
+    "# List all documents in the collection\n",
+    "try:\n",
+    "    # Query to fetch all documents of this memory type\n",
+    "    query_str = f\"SELECT META().id, * FROM `{storage.bucket_name}`.`{storage.scope_name}`.`{storage.collection_name}` WHERE memory_type = $type\"\n",
+    "    query_result = storage.cluster.query(query_str, type=storage.type)\n",
+    "    \n",
+    "    print(f\"\\nAll memory entries in Couchbase:\")\n",
+    "    print(\"-\" * 80)\n",
+    "    for i, row in enumerate(query_result, 1):\n",
+    "        doc_id = row.get('id')\n",
+    "        memory_id = row.get(storage.collection_name, {}).get('memory_id', 'unknown')\n",
+    "        content = row.get(storage.collection_name, {}).get('text', '')[:100] + \"...\"  # Truncate for readability\n",
+    "        \n",
+    "        print(f\"Entry {i}: {memory_id}\")\n",
+    "        print(f\"Content: {content}\")\n",
+    "        print(\"-\" * 80)\n",
+    "except Exception as e:\n",
+    "    print(f\"Failed to list memory entries: {str(e)}\")\n",
+    "\n",
+    "# Test memory retention\n",
+    "memory_query = \"What is Manchester City's current form according to Guardiola?\"\n",
+    "memory_results = storage.search(\n",
+    "    query=memory_query,\n",
+    "    limit=5,  # Increased to see more results\n",
+    "    score_threshold=0.0  # Lower threshold to see all results\n",
+    ")\n",
+    "\n",
+    "print(\"\\nMemory Search Results:\")\n",
+    "print(\"-\" * 80)\n",
+    "for result in memory_results:\n",
+    "    print(f\"Context: {result['context']}\")\n",
+    "    print(f\"Distance: {result['distance']} (lower = more similar)\")\n",
+    "    print(\"-\" * 80)\n",
+    "\n",
+    "# Try a more specific query to find agent interactions\n",
+    "interaction_query = \"Manchester City playing style analysis tactical\"\n",
+    "interaction_results = storage.search(\n",
+    "    query=interaction_query,\n",
+    "    limit=3,\n",
+    "    score_threshold=0.0\n",
+    ")\n",
+    "\n",
+    "print(\"\\nAgent Interaction Memory Results:\")\n",
+    "print(\"-\" * 80)\n",
+    "if interaction_results:\n",
+    "    for result in interaction_results:\n",
+    "        print(f\"Context: {result['context'][:200]}...\")  # Limit output size\n",
+    "        print(f\"Distance: {result['distance']} (lower = more similar)\")\n",
+    "        print(\"-\" * 80)\n",
+    "else:\n",
+    "    print(\"No interaction memories found. This is normal if agents haven't completed tasks yet.\")\n",
+    "    print(\"-\" * 80)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d23b2fbe",
+   "metadata": {},
+   "source": [
+    "## Conclusion"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d21915e5",
+   "metadata": {},
+   "source": [
+    "You've successfully implemented a custom memory backend for CrewAI agents using Couchbase Hyperscale and Composite Vector Indexes!"
+   ]
+  }
+ ],
+ "metadata": {
+  "jupytext": {
+   "cell_metadata_filter": "-all",
+   "main_language": "python",
+   "notebook_metadata_filter": "-all"
+  },
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/crewai-short-term-memory/query_based/frontmatter.md b/crewai-short-term-memory/query_based/frontmatter.md
new file mode 100644
index 00000000..bca09c23
--- /dev/null
+++ b/crewai-short-term-memory/query_based/frontmatter.md
@@ -0,0 +1,25 @@
+---
+# frontmatter
+path: "/tutorial-crewai-short-term-memory-couchbase-with-hyperscale-or-composite-vector-index"
+alt_paths: ["/tutorial-crewai-short-term-memory-couchbase-with-hyperscale-vector-index", "/tutorial-crewai-short-term-memory-couchbase-with-composite-vector-index"]
+title: Implementing Short-Term Memory for CrewAI Agents with Couchbase Hyperscale and Composite Vector Index
+short_title: CrewAI Short-Term Memory with Couchbase Hyperscale and Composite Vector Index
+description:
+  - Learn how to implement short-term memory for CrewAI agents using Couchbase's vector search capabilities with Hyperscale and Composite Vector Indexes.
+  - This tutorial demonstrates how to store and retrieve agent interactions using semantic search with high-performance GSI vector indexes.
+  - You'll understand how to enhance CrewAI agents with memory capabilities using LangChain and Couchbase.
+content_type: tutorial
+filter: sdk
+technology:
+  - vector search
+tags:
+  - Hyperscale Vector Index
+  - Composite Vector Index
+  - Artificial Intelligence
+  - LangChain
+  - CrewAI
+sdk_language:
+  - python
+length: 45 Mins
+---
+
diff --git a/crewai-short-term-memory/gsi/.env.sample b/crewai-short-term-memory/search_based/.env.sample
similarity index 100%
rename from crewai-short-term-memory/gsi/.env.sample
rename to crewai-short-term-memory/search_based/.env.sample
diff --git a/crewai-short-term-memory/fts/CouchbaseStorage_Demo.ipynb b/crewai-short-term-memory/search_based/CouchbaseStorage_Demo.ipynb
similarity index 97%
rename from crewai-short-term-memory/fts/CouchbaseStorage_Demo.ipynb
rename to crewai-short-term-memory/search_based/CouchbaseStorage_Demo.ipynb
index de6676af..438b1bec 100644
--- a/crewai-short-term-memory/fts/CouchbaseStorage_Demo.ipynb
+++ b/crewai-short-term-memory/search_based/CouchbaseStorage_Demo.ipynb
@@ -4,23 +4,22 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "# CrewAI with Couchbase Short-Term Memory\n",
+        "## Introduction\n",
         "\n",
-        "This notebook demonstrates how to implement a custom storage backend for CrewAI's memory system using Couchbase and vector search. Alternatively if you want to perform semantic search using the GSI index, please take a look at [this.](https://developer.couchbase.com/tutorial-crewai-short-term-memory-couchbase-with-global-secondary-index)\n",
+        "This notebook demonstrates how to implement a custom storage backend for CrewAI's memory system using Couchbase and Search Vector Index. Alternatively if you want to perform semantic search using Hyperscale or Composite Vector Index, please take a look at [this tutorial.](https://developer.couchbase.com/tutorial-crewai-short-term-memory-couchbase-with-hyperscale-or-composite-vector-index)\n",
         "\n",
         "Here's a breakdown of each section:\n",
         "\n",
-        "How to run this tutorial\n",
-        "----------------------\n",
+        "### How to run this tutorial\n",
+        "\n",
         "This tutorial is available as a Jupyter Notebook (.ipynb file) that you can run \n",
-        "interactively. You can access the original notebook [here](https://github.com/couchbase-examples/vector-search-cookbook/blob/main/crewai-short-term-memory/fts/CouchbaseStorage_Demo.ipynb).\n",
+        "interactively. You can access the original notebook [here](https://github.com/couchbase-examples/vector-search-cookbook/blob/main/crewai-short-term-memory/search_based/CouchbaseStorage_Demo.ipynb).\n",
         "\n",
         "You can either:\n",
         "- Download the notebook file and run it on [Google Colab](https://colab.research.google.com)\n",
         "- Run it on your system by setting up the Python environment\n",
         "\n",
-        "Before you start\n",
-        "---------------\n",
+        "### Before you start\n",
         "\n",
         "1. Create and Deploy Your Free Tier Operational cluster on [Capella](https://cloud.couchbase.com/sign-up)\n",
         "   - To get started with [Couchbase Capella](https://cloud.couchbase.com), create an account and use it to deploy \n",
@@ -107,7 +106,7 @@
     },
     {
       "cell_type": "code",
-      "execution_count": 1,
+      "execution_count": null,
       "metadata": {},
       "outputs": [
         {
@@ -119,7 +118,7 @@
         }
       ],
       "source": [
-        "%pip install --quiet crewai==0.186.1 langchain-couchbase==0.4.0 langchain-openai==0.3.33 python-dotenv==1.1.1"
+        "%pip install --quiet crewai==0.186.1 langchain-couchbase==1.0.1 langchain-openai python-dotenv==1.1.1"
       ]
     },
     {
@@ -183,7 +182,24 @@
         "> - `COLLECTION_NAME` (optional): Collection name (defaults to \"crew\")\n",
         "> - `INDEX_NAME` (optional): Vector search index name (defaults to \"vector_search_crew\")\n",
         ">\n",
-        "> You can set these variables in a `.env` file in the same directory as this notebook, or set them directly in your environment."
+        "> You can set these variables in a `.env` file in the same directory as this notebook, or set them directly in your environment.\n",
+        "\n",
+        "### Google Colab Setup\n",
+        "\n",
+        "If running in Google Colab, use the Secrets feature:\n",
+        "1. Click the 🔑 (key icon) in the left sidebar\n",
+        "2. Add secrets with these names: `OPENAI_API_KEY`, `CB_HOST`, `CB_USERNAME`, `CB_PASSWORD`\n",
+        "3. Toggle \"Notebook access\" for each secret\n",
+        "4. Run this code before the environment verification cell:\n",
+        "\n",
+        "```python\n",
+        "from google.colab import userdata\n",
+        "import os\n",
+        "os.environ['OPENAI_API_KEY'] = userdata.get('OPENAI_API_KEY')\n",
+        "os.environ['CB_HOST'] = userdata.get('CB_HOST')\n",
+        "os.environ['CB_USERNAME'] = userdata.get('CB_USERNAME')\n",
+        "os.environ['CB_PASSWORD'] = userdata.get('CB_PASSWORD')\n",
+        "```"
       ]
     },
     {
@@ -207,17 +223,18 @@
       "source": [
         "## Implement CouchbaseStorage\n",
         "\n",
-        "This section demonstrates the implementation of a custom vector storage solution using Couchbase:\n",
+        "This section demonstrates the implementation of a custom vector storage solution using Couchbase with Search Vector Index:\n",
         "\n",
         "> **Note on Implementation:** This example uses the LangChain Couchbase integration (`langchain_couchbase`) for simplicity and to demonstrate integration with the broader LangChain ecosystem. In production environments, you may want to use the Couchbase SDK directly for better performance and more control.\n",
         "\n",
-        "> For more information on using the Couchbase SDK directly, refer to:\n",
-        "> - [Couchbase Python SDK Documentation](https://docs.couchbase.com/python-sdk/current/howtos/full-text-searching-with-sdk.html#single-vector-query)"
+        "> For more information on using the Couchbase SDK directly and Search Vector Indexes, refer to:\n",
+        "> - [Couchbase Python SDK Documentation](https://docs.couchbase.com/python-sdk/current/howtos/full-text-searching-with-sdk.html#single-vector-query)\n",
+        "> - [Couchbase Vector Index Documentation](https://docs.couchbase.com/cloud/vector-index/use-vector-indexes.html)"
       ]
     },
     {
       "cell_type": "code",
-      "execution_count": 4,
+      "execution_count": null,
       "metadata": {},
       "outputs": [],
       "source": [
@@ -398,11 +415,11 @@
         "                            break\n",
         "                    break\n",
         "            if not search_available:\n",
-        "                raise RuntimeError(\"Search/FTS service not found or not responding\")\n",
+        "                raise RuntimeError(\"Search service not found or not responding\")\n",
         "            \n",
         "            # Set up storage configuration\n",
         "            self.bucket_name = os.getenv('CB_BUCKET_NAME', 'vector-search-testing')\n",
-        "            self.scope_name = os.getenv('SCOPE_NAME', 'shared')\n",
+        "            self.scope_name = os.getenv('CB_SCOPE_NAME', 'shared')\n",
         "            self.collection_name = os.getenv('COLLECTION_NAME', 'crew')\n",
         "            self.index_name = os.getenv('INDEX_NAME', 'vector_search_crew')\n",
         "\n",
diff --git a/crewai-short-term-memory/fts/crew_index.json b/crewai-short-term-memory/search_based/crew_index.json
similarity index 99%
rename from crewai-short-term-memory/fts/crew_index.json
rename to crewai-short-term-memory/search_based/crew_index.json
index 0409dc36..2ad38a04 100644
--- a/crewai-short-term-memory/fts/crew_index.json
+++ b/crewai-short-term-memory/search_based/crew_index.json
@@ -71,3 +71,4 @@
     },
     "sourceParams": {}
 }
+
diff --git a/crewai-short-term-memory/fts/frontmatter.md b/crewai-short-term-memory/search_based/frontmatter.md
similarity index 66%
rename from crewai-short-term-memory/fts/frontmatter.md
rename to crewai-short-term-memory/search_based/frontmatter.md
index 861c9c6e..6450a510 100644
--- a/crewai-short-term-memory/fts/frontmatter.md
+++ b/crewai-short-term-memory/search_based/frontmatter.md
@@ -1,10 +1,10 @@
 ---
 # frontmatter
-path: "/tutorial-crewai-short-term-memory-couchbase-with-fts"
-title: Implementing Short-Term Memory for CrewAI Agents with Couchbase using FTS Service
-short_title: CrewAI Short-Term Memory with Couchbase using FTS
+path: "/tutorial-crewai-short-term-memory-couchbase-with-search-vector-index"
+title: Implementing Short-Term Memory for CrewAI Agents with Couchbase Search Vector Index
+short_title: CrewAI Short-Term Memory with Couchbase Search Vector Index
 description:
-  - Learn how to implement short-term memory for CrewAI agents using Couchbase's vector search capabilities using FTS.
+  - Learn how to implement short-term memory for CrewAI agents using Couchbase's vector search capabilities with Search Vector Index.
   - This tutorial demonstrates how to store and retrieve agent interactions using semantic search.
   - You'll understand how to enhance CrewAI agents with memory capabilities using LangChain and Couchbase.
 content_type: tutorial
@@ -12,7 +12,7 @@ filter: sdk
 technology:
   - vector search
 tags:
-  - FTS
+  - Search Vector Index
   - Artificial Intelligence
   - LangChain
   - CrewAI
@@ -20,3 +20,4 @@ sdk_language:
   - python
 length: 45 Mins
 ---
+

From 5823b444104e3560467b25262bda84a541dad329 Mon Sep 17 00:00:00 2001
From: Dhiraj Kumar Azad 
Date: Thu, 11 Dec 2025 17:27:18 +0530
Subject: [PATCH 2/6] change for failed CI

---
 crewai-short-term-memory/query_based/frontmatter.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/crewai-short-term-memory/query_based/frontmatter.md b/crewai-short-term-memory/query_based/frontmatter.md
index bca09c23..0b281a03 100644
--- a/crewai-short-term-memory/query_based/frontmatter.md
+++ b/crewai-short-term-memory/query_based/frontmatter.md
@@ -2,7 +2,7 @@
 # frontmatter
 path: "/tutorial-crewai-short-term-memory-couchbase-with-hyperscale-or-composite-vector-index"
 alt_paths: ["/tutorial-crewai-short-term-memory-couchbase-with-hyperscale-vector-index", "/tutorial-crewai-short-term-memory-couchbase-with-composite-vector-index"]
-title: Implementing Short-Term Memory for CrewAI Agents with Couchbase Hyperscale and Composite Vector Index
+title: Implement Short-Term Memory for CrewAI Agents with Couchbase Hyperscale and Composite Vector Index
 short_title: CrewAI Short-Term Memory with Couchbase Hyperscale and Composite Vector Index
 description:
   - Learn how to implement short-term memory for CrewAI agents using Couchbase's vector search capabilities with Hyperscale and Composite Vector Indexes.

From e1f2f1e55051967a8f8289dd7adc0522d9a59d77 Mon Sep 17 00:00:00 2001
From: Dhiraj Kumar Azad 
Date: Thu, 11 Dec 2025 17:32:37 +0530
Subject: [PATCH 3/6] Shortened the length of short title

---
 crewai-short-term-memory/query_based/frontmatter.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/crewai-short-term-memory/query_based/frontmatter.md b/crewai-short-term-memory/query_based/frontmatter.md
index 0b281a03..e487ed93 100644
--- a/crewai-short-term-memory/query_based/frontmatter.md
+++ b/crewai-short-term-memory/query_based/frontmatter.md
@@ -3,7 +3,7 @@
 path: "/tutorial-crewai-short-term-memory-couchbase-with-hyperscale-or-composite-vector-index"
 alt_paths: ["/tutorial-crewai-short-term-memory-couchbase-with-hyperscale-vector-index", "/tutorial-crewai-short-term-memory-couchbase-with-composite-vector-index"]
 title: Implement Short-Term Memory for CrewAI Agents with Couchbase Hyperscale and Composite Vector Index
-short_title: CrewAI Short-Term Memory with Couchbase Hyperscale and Composite Vector Index
+short_title: CrewAI Short-Term Memory with Hyperscale & Composite Index
 description:
   - Learn how to implement short-term memory for CrewAI agents using Couchbase's vector search capabilities with Hyperscale and Composite Vector Indexes.
   - This tutorial demonstrates how to store and retrieve agent interactions using semantic search with high-performance GSI vector indexes.

From d72074eb4b5dac80584056a2f3296480693d959e Mon Sep 17 00:00:00 2001
From: Dhiraj Kumar Azad 
Date: Thu, 11 Dec 2025 17:41:54 +0530
Subject: [PATCH 4/6] minor changes

---
 .../query_based/CouchbaseStorage_Demo.ipynb                   | 4 ++--
 crewai-short-term-memory/query_based/frontmatter.md           | 2 +-
 .../search_based/CouchbaseStorage_Demo.ipynb                  | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb b/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb
index 9ace0da1..13ed227b 100644
--- a/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb
+++ b/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb
@@ -564,8 +564,8 @@
     "            # Set up storage configuration\n",
     "            self.bucket_name = os.getenv('CB_BUCKET_NAME', 'vector-search-testing')\n",
     "            self.scope_name = os.getenv('CB_SCOPE_NAME', 'shared')\n",
-    "            self.collection_name = os.getenv('COLLECTION_NAME', 'crew')\n",
-    "            self.index_name = os.getenv('INDEX_NAME', 'vector_search_crew_gsi')\n",
+    "            self.collection_name = os.getenv('CB_COLLECTION_NAME', 'crew')\n",
+    "            self.index_name = os.getenv('CB_INDEX_NAME', 'vector_search_crew_gsi')\n",
     "\n",
     "            # Initialize vector store\n",
     "            self.vector_store = CouchbaseQueryVectorStore(\n",
diff --git a/crewai-short-term-memory/query_based/frontmatter.md b/crewai-short-term-memory/query_based/frontmatter.md
index e487ed93..ac3ebd48 100644
--- a/crewai-short-term-memory/query_based/frontmatter.md
+++ b/crewai-short-term-memory/query_based/frontmatter.md
@@ -6,7 +6,7 @@ title: Implement Short-Term Memory for CrewAI Agents with Couchbase Hyperscale a
 short_title: CrewAI Short-Term Memory with Hyperscale & Composite Index
 description:
   - Learn how to implement short-term memory for CrewAI agents using Couchbase's vector search capabilities with Hyperscale and Composite Vector Indexes.
-  - This tutorial demonstrates how to store and retrieve agent interactions using semantic search with high-performance GSI vector indexes.
+  - This tutorial demonstrates how to store and retrieve agent interactions using semantic search with high-performance Hyperscale and Composite vector indexes.
   - You'll understand how to enhance CrewAI agents with memory capabilities using LangChain and Couchbase.
 content_type: tutorial
 filter: sdk
diff --git a/crewai-short-term-memory/search_based/CouchbaseStorage_Demo.ipynb b/crewai-short-term-memory/search_based/CouchbaseStorage_Demo.ipynb
index 438b1bec..30e0b44d 100644
--- a/crewai-short-term-memory/search_based/CouchbaseStorage_Demo.ipynb
+++ b/crewai-short-term-memory/search_based/CouchbaseStorage_Demo.ipynb
@@ -420,8 +420,8 @@
         "            # Set up storage configuration\n",
         "            self.bucket_name = os.getenv('CB_BUCKET_NAME', 'vector-search-testing')\n",
         "            self.scope_name = os.getenv('CB_SCOPE_NAME', 'shared')\n",
-        "            self.collection_name = os.getenv('COLLECTION_NAME', 'crew')\n",
-        "            self.index_name = os.getenv('INDEX_NAME', 'vector_search_crew')\n",
+        "            self.collection_name = os.getenv('CB_COLLECTION_NAME', 'crew')\n",
+        "            self.index_name = os.getenv('CB_INDEX_NAME', 'vector_search_crew')\n",
         "\n",
         "            # Initialize vector store\n",
         "            self.vector_store = CouchbaseSearchVectorStore(\n",

From 4332dbd48e938ee0f11378149f3c7c63e336d399 Mon Sep 17 00:00:00 2001
From: Dhiraj Kumar Azad 
Date: Tue, 16 Dec 2025 10:30:15 +0530
Subject: [PATCH 5/6] Addressed comments

---
 .../query_based/CouchbaseStorage_Demo.ipynb   | 89 ++++++++++++++++---
 .../search_based/CouchbaseStorage_Demo.ipynb  |  2 +-
 2 files changed, 80 insertions(+), 11 deletions(-)

diff --git a/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb b/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb
index 13ed227b..13a807ea 100644
--- a/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb
+++ b/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb
@@ -5,7 +5,26 @@
    "id": "fa3af5ad",
    "metadata": {},
    "source": [
-    "## Introduction"
+    "## Introduction\n",
+    "\n",
+    "This notebook demonstrates how to implement a custom storage backend for CrewAI's memory system using Couchbase with Hyperscale or Composite Vector Indexes. These indexes leverage Couchbase's Query service for high-performance vector search, making them ideal for AI agent memory systems that require scalability and efficient semantic retrieval.\n",
+    "\n",
+    "If you prefer to use Search-based vector indexes instead, check out [this tutorial](https://developer.couchbase.com/tutorial-crewai-short-term-memory-couchbase-with-search-vector-index).\n",
+    "\n",
+    "### How to Run This Tutorial\n",
+    "\n",
+    "This tutorial is available as a Jupyter Notebook (.ipynb file) that you can run interactively. You can access the original notebook [here](https://github.com/couchbase-examples/vector-search-cookbook/blob/main/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb).\n",
+    "\n",
+    "You can either:\n",
+    "- Download the notebook file and run it on [Google Colab](https://colab.research.google.com)\n",
+    "- Run it on your system by setting up the Python environment\n",
+    "\n",
+    "### What You'll Learn\n",
+    "\n",
+    "- How to create a custom `CouchbaseRAGStorage` class that integrates with CrewAI's memory system\n",
+    "- Using Couchbase Hyperscale Vector Indexes for semantic search\n",
+    "- Configuring OpenAI embeddings for vector representation\n",
+    "- Building AI agents with persistent memory capabilities"
    ]
   },
   {
@@ -206,7 +225,7 @@
    "id": "bd67cca9",
    "metadata": {},
    "source": [
-    "Import libraries for CrewAI memory storage, Couchbase GSI vector search, and OpenAI embeddings."
+    "Import libraries for CrewAI memory storage, Couchbase vector search, and OpenAI embeddings."
    ]
   },
   {
@@ -565,7 +584,7 @@
     "            self.bucket_name = os.getenv('CB_BUCKET_NAME', 'vector-search-testing')\n",
     "            self.scope_name = os.getenv('CB_SCOPE_NAME', 'shared')\n",
     "            self.collection_name = os.getenv('CB_COLLECTION_NAME', 'crew')\n",
-    "            self.index_name = os.getenv('CB_INDEX_NAME', 'vector_search_crew_gsi')\n",
+    "            self.index_name = os.getenv('CB_INDEX_NAME', 'vector_search_crew_hyperscale')\n",
     "\n",
     "            # Initialize vector store\n",
     "            self.vector_store = CouchbaseQueryVectorStore(\n",
@@ -700,7 +719,19 @@
     "test_query = \"What did Guardiola say about Manchester City?\"\n",
     "print(\"Testing baseline memory search performance without vector index optimization...\")\n",
     "baseline_time = test_memory_search_performance(storage, test_query, \"Baseline Search\")\n",
-    "print(f\"\\nBaseline memory search time (without vector index): {baseline_time:.4f} seconds\\n\")"
+    "print(f\"\\nBaseline memory search time (without vector index): {baseline_time:.4f} seconds\\n\")\n",
+    "\n",
+    "# Output:\n",
+    "# Testing baseline memory search performance without vector index optimization...\n",
+    "#\n",
+    "# [Baseline Search] Testing memory search performance\n",
+    "# [Baseline Search] Query: 'What did Guardiola say about Manchester City?'\n",
+    "# [Baseline Search] Memory search completed in 0.6159 seconds\n",
+    "# [Baseline Search] Found 1 memories\n",
+    "# [Baseline Search] Top result distance: 0.340130 (lower = more similar)\n",
+    "# [Baseline Search] Top result preview: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a ...\n",
+    "#\n",
+    "# Baseline memory search time (without vector index): 0.6159 seconds"
    ]
   },
   {
@@ -745,7 +776,12 @@
     "    if \"already exists\" in str(e).lower():\n",
     "        print(f\"Vector index '{storage.index_name}' already exists, proceeding...\")\n",
     "    else:\n",
-    "        print(f\"Error creating vector index: {str(e)}\")"
+    "        print(f\"Error creating vector index: {str(e)}\")\n",
+    "\n",
+    "# Output:\n",
+    "# Creating Hyperscale Vector Index...\n",
+    "# Hyperscale Vector Index created successfully: vector_search_crew\n",
+    "# Waiting for index to become available..."
    ]
   },
   {
@@ -766,7 +802,7 @@
     "```python\n",
     "# Alternative: Create a Composite Vector Index for filtered memory searches\n",
     "storage.vector_store.create_index(\n",
-    "    index_type=IndexType.COMPOSITE,  # Instead of IndexType.BHIVE\n",
+    "    index_type=IndexType.COMPOSITE,  # Instead of IndexType.HYPERSCALE\n",
     "    index_name=storage.index_name,\n",
     "    index_description=\"IVF,SQ8\"      # Same quantization settings\n",
     ")\n",
@@ -786,7 +822,7 @@
    "id": "5d786f04",
    "metadata": {},
    "source": [
-    "Test the same memory search with Hyperscale Vector Index (BHIVE) optimization."
+    "Test the same memory search with Hyperscale Vector Index optimization."
    ]
   },
   {
@@ -798,7 +834,17 @@
    "source": [
     "# Test memory search performance with Hyperscale Vector Index\n",
     "print(\"Testing memory search performance with Hyperscale Vector Index optimization...\")\n",
-    "optimized_time = test_memory_search_performance(storage, test_query, \"Vector Index-Optimized Search\")"
+    "optimized_time = test_memory_search_performance(storage, test_query, \"Vector Index-Optimized Search\")\n",
+    "\n",
+    "# Output:\n",
+    "# Testing memory search performance with Hyperscale Vector Index optimization...\n",
+    "#\n",
+    "# [Vector Index-Optimized Search] Testing memory search performance\n",
+    "# [Vector Index-Optimized Search] Query: 'What did Guardiola say about Manchester City?'\n",
+    "# [Vector Index-Optimized Search] Memory search completed in 0.5910 seconds\n",
+    "# [Vector Index-Optimized Search] Found 1 memories\n",
+    "# [Vector Index-Optimized Search] Top result distance: 0.340142 (lower = more similar)\n",
+    "# [Vector Index-Optimized Search] Top result preview: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a ..."
    ]
   },
   {
@@ -918,7 +964,30 @@
     "print(f\"• Performance gains are most dramatic for complex semantic memory queries\")\n",
     "print(f\"• Hyperscale optimization is particularly effective for agent conversational memory\")\n",
     "print(f\"• Combined with proper quantization (SQ8), vector indexes deliver production-ready performance\")\n",
-    "print(f\"• These performance improvements directly benefit agent response times and scalability\")"
+    "print(f\"• These performance improvements directly benefit agent response times and scalability\")\n",
+    "\n",
+    "# Output:\n",
+    "# ================================================================================\n",
+    "# MEMORY SEARCH PERFORMANCE OPTIMIZATION SUMMARY\n",
+    "# ================================================================================\n",
+    "# Phase 1 - Baseline Search (No Vector Index):     0.6159 seconds\n",
+    "# Phase 2 - Vector Index-Optimized Search:         0.5910 seconds\n",
+    "# Phase 3 - Cache Benefits:\n",
+    "#   First execution (cache miss):         0.6076 seconds\n",
+    "#   Second execution (cache hit):         0.4745 seconds\n",
+    "#\n",
+    "# --------------------------------------------------------------------------------\n",
+    "# MEMORY SEARCH OPTIMIZATION IMPACT:\n",
+    "# --------------------------------------------------------------------------------\n",
+    "# Vector Index Benefit:      1.04x faster (4.0% improvement)\n",
+    "# Cache Benefit:          1.28x faster (21.9% improvement)\n",
+    "#\n",
+    "# Key Insights for Agent Memory Performance:\n",
+    "# • Hyperscale Vector Indexes provide significant performance improvements for memory search\n",
+    "# • Performance gains are most dramatic for complex semantic memory queries\n",
+    "# • Hyperscale optimization is particularly effective for agent conversational memory\n",
+    "# • Combined with proper quantization (SQ8), vector indexes deliver production-ready performance\n",
+    "# • These performance improvements directly benefit agent response times and scalability"
    ]
   },
   {
@@ -926,7 +995,7 @@
    "id": "c4b069f8",
    "metadata": {},
    "source": [
-    "**Note on Hyperscale Vector Index Performance:** The Hyperscale Vector Index (BHIVE) may show slower performance for very small datasets (few documents) due to the additional overhead of maintaining the index structure. However, as the dataset scales up, the Hyperscale Vector Index becomes significantly faster than traditional vector searches. The initial overhead investment pays off dramatically with larger memory stores, making it essential for production agent deployments with substantial conversational history."
+    "**Note on Hyperscale Vector Index Performance:** The Hyperscale Vector Index may show slower performance for very small datasets (few documents) due to the additional overhead of maintaining the index structure. However, as the dataset scales up, the Hyperscale Vector Index becomes significantly faster than traditional vector searches. The initial overhead investment pays off dramatically with larger memory stores, making it essential for production agent deployments with substantial conversational history."
    ]
   },
   {
diff --git a/crewai-short-term-memory/search_based/CouchbaseStorage_Demo.ipynb b/crewai-short-term-memory/search_based/CouchbaseStorage_Demo.ipynb
index 30e0b44d..f1ea3516 100644
--- a/crewai-short-term-memory/search_based/CouchbaseStorage_Demo.ipynb
+++ b/crewai-short-term-memory/search_based/CouchbaseStorage_Demo.ipynb
@@ -39,7 +39,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "# Memory in AI Agents\n",
+        "## Memory in AI Agents\n",
         "\n",
         "Memory in AI agents is a crucial capability that allows them to retain and utilize information across interactions, making them more effective and contextually aware. Without memory, agents would be limited to processing only the immediate input, lacking the ability to build upon past experiences or maintain continuity in conversations.\n",
         "\n",

From 7818a829b149ebb88542c5523debc06867fca006 Mon Sep 17 00:00:00 2001
From: Dhiraj Kumar Azad 
Date: Wed, 17 Dec 2025 11:36:25 +0530
Subject: [PATCH 6/6] Addressed comments

---
 .../query_based/CouchbaseStorage_Demo.ipynb   | 146 ++++++++++--------
 1 file changed, 84 insertions(+), 62 deletions(-)

diff --git a/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb b/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb
index 13a807ea..0a48ea46 100644
--- a/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb
+++ b/crewai-short-term-memory/query_based/CouchbaseStorage_Demo.ipynb
@@ -710,28 +710,34 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 14,
    "id": "383bb87d",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Testing baseline memory search performance without vector index optimization...\n",
+      "\n",
+      "[Baseline Search] Testing memory search performance\n",
+      "[Baseline Search] Query: 'What did Guardiola say about Manchester City?'\n",
+      "[Baseline Search] Memory search completed in 0.6159 seconds\n",
+      "[Baseline Search] Found 1 memories\n",
+      "[Baseline Search] Top result distance: 0.340130 (lower = more similar)\n",
+      "[Baseline Search] Top result preview: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a ...\n",
+      "\n",
+      "Baseline memory search time (without vector index): 0.6159 seconds\n",
+      "\n"
+     ]
+    }
+   ],
    "source": [
     "# Test baseline memory search performance without vector index\n",
     "test_query = \"What did Guardiola say about Manchester City?\"\n",
     "print(\"Testing baseline memory search performance without vector index optimization...\")\n",
     "baseline_time = test_memory_search_performance(storage, test_query, \"Baseline Search\")\n",
-    "print(f\"\\nBaseline memory search time (without vector index): {baseline_time:.4f} seconds\\n\")\n",
-    "\n",
-    "# Output:\n",
-    "# Testing baseline memory search performance without vector index optimization...\n",
-    "#\n",
-    "# [Baseline Search] Testing memory search performance\n",
-    "# [Baseline Search] Query: 'What did Guardiola say about Manchester City?'\n",
-    "# [Baseline Search] Memory search completed in 0.6159 seconds\n",
-    "# [Baseline Search] Found 1 memories\n",
-    "# [Baseline Search] Top result distance: 0.340130 (lower = more similar)\n",
-    "# [Baseline Search] Top result preview: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a ...\n",
-    "#\n",
-    "# Baseline memory search time (without vector index): 0.6159 seconds"
+    "print(f\"\\nBaseline memory search time (without vector index): {baseline_time:.4f} seconds\\n\")"
    ]
   },
   {
@@ -752,10 +758,20 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 15,
    "id": "bde97a46",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Creating Hyperscale Vector Index...\n",
+      "Hyperscale Vector Index created successfully: vector_search_crew_hyperscale\n",
+      "Waiting for index to become available...\n"
+     ]
+    }
+   ],
    "source": [
     "# Create Hyperscale Vector Index for optimal performance\n",
     "print(\"Creating Hyperscale Vector Index...\")\n",
@@ -776,12 +792,7 @@
     "    if \"already exists\" in str(e).lower():\n",
     "        print(f\"Vector index '{storage.index_name}' already exists, proceeding...\")\n",
     "    else:\n",
-    "        print(f\"Error creating vector index: {str(e)}\")\n",
-    "\n",
-    "# Output:\n",
-    "# Creating Hyperscale Vector Index...\n",
-    "# Hyperscale Vector Index created successfully: vector_search_crew\n",
-    "# Waiting for index to become available..."
+    "        print(f\"Error creating vector index: {str(e)}\")"
    ]
   },
   {
@@ -827,24 +838,29 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 16,
    "id": "849758ae",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Testing memory search performance with Hyperscale Vector Index optimization...\n",
+      "\n",
+      "[Vector Index-Optimized Search] Testing memory search performance\n",
+      "[Vector Index-Optimized Search] Query: 'What did Guardiola say about Manchester City?'\n",
+      "[Vector Index-Optimized Search] Memory search completed in 0.5910 seconds\n",
+      "[Vector Index-Optimized Search] Found 1 memories\n",
+      "[Vector Index-Optimized Search] Top result distance: 0.340142 (lower = more similar)\n",
+      "[Vector Index-Optimized Search] Top result preview: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a ...\n"
+     ]
+    }
+   ],
    "source": [
     "# Test memory search performance with Hyperscale Vector Index\n",
     "print(\"Testing memory search performance with Hyperscale Vector Index optimization...\")\n",
-    "optimized_time = test_memory_search_performance(storage, test_query, \"Vector Index-Optimized Search\")\n",
-    "\n",
-    "# Output:\n",
-    "# Testing memory search performance with Hyperscale Vector Index optimization...\n",
-    "#\n",
-    "# [Vector Index-Optimized Search] Testing memory search performance\n",
-    "# [Vector Index-Optimized Search] Query: 'What did Guardiola say about Manchester City?'\n",
-    "# [Vector Index-Optimized Search] Memory search completed in 0.5910 seconds\n",
-    "# [Vector Index-Optimized Search] Found 1 memories\n",
-    "# [Vector Index-Optimized Search] Top result distance: 0.340142 (lower = more similar)\n",
-    "# [Vector Index-Optimized Search] Top result preview: Pep Guardiola praised Manchester City's current form, saying 'The team is playing well, we are in a ..."
+    "optimized_time = test_memory_search_performance(storage, test_query, \"Vector Index-Optimized Search\")"
    ]
   },
   {
@@ -924,10 +940,39 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 18,
    "id": "f813eb1a",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "================================================================================\n",
+      "MEMORY SEARCH PERFORMANCE OPTIMIZATION SUMMARY\n",
+      "================================================================================\n",
+      "Phase 1 - Baseline Search (No Vector Index):     0.6159 seconds\n",
+      "Phase 2 - Vector Index-Optimized Search:         0.5910 seconds\n",
+      "Phase 3 - Cache Benefits:\n",
+      "  First execution (cache miss):         0.6076 seconds\n",
+      "  Second execution (cache hit):         0.4745 seconds\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "MEMORY SEARCH OPTIMIZATION IMPACT:\n",
+      "--------------------------------------------------------------------------------\n",
+      "Vector Index Benefit:      1.04x faster (4.0% improvement)\n",
+      "Cache Benefit:          1.28x faster (21.9% improvement)\n",
+      "\n",
+      "Key Insights for Agent Memory Performance:\n",
+      "• Hyperscale Vector Indexes provide significant performance improvements for memory search\n",
+      "• Performance gains are most dramatic for complex semantic memory queries\n",
+      "• Hyperscale optimization is particularly effective for agent conversational memory\n",
+      "• Combined with proper quantization (SQ8), vector indexes deliver production-ready performance\n",
+      "• These performance improvements directly benefit agent response times and scalability\n"
+     ]
+    }
+   ],
    "source": [
     "print(\"\\n\" + \"=\"*80)\n",
     "print(\"MEMORY SEARCH PERFORMANCE OPTIMIZATION SUMMARY\")\n",
@@ -964,30 +1009,7 @@
     "print(f\"• Performance gains are most dramatic for complex semantic memory queries\")\n",
     "print(f\"• Hyperscale optimization is particularly effective for agent conversational memory\")\n",
     "print(f\"• Combined with proper quantization (SQ8), vector indexes deliver production-ready performance\")\n",
-    "print(f\"• These performance improvements directly benefit agent response times and scalability\")\n",
-    "\n",
-    "# Output:\n",
-    "# ================================================================================\n",
-    "# MEMORY SEARCH PERFORMANCE OPTIMIZATION SUMMARY\n",
-    "# ================================================================================\n",
-    "# Phase 1 - Baseline Search (No Vector Index):     0.6159 seconds\n",
-    "# Phase 2 - Vector Index-Optimized Search:         0.5910 seconds\n",
-    "# Phase 3 - Cache Benefits:\n",
-    "#   First execution (cache miss):         0.6076 seconds\n",
-    "#   Second execution (cache hit):         0.4745 seconds\n",
-    "#\n",
-    "# --------------------------------------------------------------------------------\n",
-    "# MEMORY SEARCH OPTIMIZATION IMPACT:\n",
-    "# --------------------------------------------------------------------------------\n",
-    "# Vector Index Benefit:      1.04x faster (4.0% improvement)\n",
-    "# Cache Benefit:          1.28x faster (21.9% improvement)\n",
-    "#\n",
-    "# Key Insights for Agent Memory Performance:\n",
-    "# • Hyperscale Vector Indexes provide significant performance improvements for memory search\n",
-    "# • Performance gains are most dramatic for complex semantic memory queries\n",
-    "# • Hyperscale optimization is particularly effective for agent conversational memory\n",
-    "# • Combined with proper quantization (SQ8), vector indexes deliver production-ready performance\n",
-    "# • These performance improvements directly benefit agent response times and scalability"
+    "print(f\"• These performance improvements directly benefit agent response times and scalability\")"
    ]
   },
   {