Developer's Catalog

Database

Redis + Vector Search

Overview

Redis is known as a high-speed in-memory data store, but with extensions like RediSearch module, RedisJSON, and RedisAI, it also provides vector search capabilities. Redis Stack (formerly Redis Enterprise) integrates these features, enabling high-performance solutions that combine real-time vector search with caching.

Details

Redis was developed by Salvatore Sanfilippo in 2009 and is now led by Redis Labs (now Redis Inc.). Vector search functionality is provided as part of the RediSearch module, supporting both flat index and HNSW algorithms. Leveraging its in-memory database characteristics, it can perform vector searches with millisecond-level low latency.

Key features of Redis vector search:

• Ultra-fast search through in-memory processing
• Support for flat index and HNSW algorithms
• Real-time index updates
• Hybrid search (vector + full-text + numeric filters)
• Multiple distance metrics (L2, inner product, cosine similarity)
• Vector range queries
• Horizontal scaling with Redis Cluster
• Persistence options (RDB, AOF)
• Real-time notifications via Pub/Sub
• Transaction support

Architecture Features

• High-speed processing with single-threaded event loop
• High availability through replication and failover
• Automatic failover with Redis Sentinel
• Memory-efficient data structures

Pros and Cons

Pros

• Ultra-fast performance: Extremely low latency through in-memory processing
• Real-time capability: Instant index updates and query responses
• Integrated solution: Cache, session management, and vector search in one system
• Rich features: Full-text search, geospatial search, time-series data, and more
• Simple operations: Single Redis instance serves multiple purposes
• Mature ecosystem: Client libraries available in many languages

Cons

• Memory cost: High cost due to keeping all data in memory
• Scalability limitations: Limited by single node memory capacity
• Persistence overhead: Performance degradation during disk persistence
• Not suitable for large datasets: Memory constraints make it unsuitable for large-scale data
• Backup complexity: Requires strategy for backing up in-memory data

Key Links

• Official Site
• Redis Stack
• RediSearch Documentation
• Vector Search Guide
• GitHub

Example Usage

Installation & Setup

# Run Redis Stack with Docker
docker run -d --name redis-stack \
  -p 6379:6379 \
  -p 8001:8001 \
  redis/redis-stack:latest

# Install Redis client (Python)
pip install redis redis-py-search

# Node.js
npm install redis

Basic Vector Search Operations

import redis
from redis.commands.search.field import VectorField, TextField, NumericField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType
from redis.commands.search.query import Query
import numpy as np

# Connection
r = redis.Redis(host='localhost', port=6379, decode_responses=True)

# Create index
schema = [
    TextField("title"),
    TextField("content"),
    VectorField("embedding", 
        "HNSW", {
            "TYPE": "FLOAT32",
            "DIM": 768,
            "DISTANCE_METRIC": "COSINE",
            "M": 16,
            "EF_CONSTRUCTION": 200
        }
    ),
    NumericField("year")
]

# Index definition
definition = IndexDefinition(
    prefix=["doc:"],
    index_type=IndexType.HASH
)

# Create index
r.ft("documents").create_index(
    fields=schema,
    definition=definition
)

# Add document
embedding = np.random.rand(768).astype(np.float32)
doc_id = "doc:1"

r.hset(doc_id, mapping={
    "title": "Redis Vector Search",
    "content": "Implementing real-time vector search with Redis",
    "embedding": embedding.tobytes(),
    "year": 2024
})

# Vector search
query_vector = np.random.rand(768).astype(np.float32)
q = Query("*=>[KNN 10 @embedding $vec AS score]")\
    .sort_by("score")\
    .return_fields("title", "content", "score")\
    .paging(0, 10)\
    .dialect(2)

results = r.ft("documents").search(
    q, 
    query_params={"vec": query_vector.tobytes()}
)

for doc in results.docs:
    print(f"Title: {doc.title}, Score: {doc.score}")

Hybrid Search

# Combined text and vector search
hybrid_query = Query(
    "(@content:Redis) => [KNN 5 @embedding $vec AS score]"
).sort_by("score").dialect(2)

results = r.ft("documents").search(
    hybrid_query,
    query_params={"vec": query_vector.tobytes()}
)

# Vector search with filtering
filtered_query = Query(
    "(@year:[2023 2024]) => [KNN 10 @embedding $vec AS score]"
).sort_by("score").dialect(2)

results = r.ft("documents").search(
    filtered_query,
    query_params={"vec": query_vector.tobytes()}
)

Batch Processing and Pipeline

# Efficient batch insertion using pipeline
pipe = r.pipeline()

for i in range(1000):
    doc_id = f"doc:{i}"
    embedding = np.random.rand(768).astype(np.float32)
    
    pipe.hset(doc_id, mapping={
        "title": f"Document {i}",
        "content": f"Content of document {i}",
        "embedding": embedding.tobytes(),
        "year": 2024
    })

# Execute batch
pipe.execute()

Real-time Updates and Pub/Sub

import threading

# Monitor vector updates
def vector_update_listener():
    pubsub = r.pubsub()
    pubsub.subscribe("vector_updates")
    
    for message in pubsub.listen():
        if message['type'] == 'message':
            print(f"Vector updated: {message['data']}")

# Start listener in separate thread
listener_thread = threading.Thread(target=vector_update_listener)
listener_thread.start()

# Update vector and notify
new_embedding = np.random.rand(768).astype(np.float32)
r.hset("doc:1", "embedding", new_embedding.tobytes())
r.publish("vector_updates", "doc:1")

Performance Optimization

# Batch queries
def batch_vector_search(query_vectors, k=10):
    results = []
    for vec in query_vectors:
        q = Query("*=>[KNN {} @embedding $vec AS score]".format(k))\
            .sort_by("score")\
            .paging(0, k)\
            .dialect(2)
        
        result = r.ft("documents").search(
            q, 
            query_params={"vec": vec.tobytes()}
        )
        results.append(result)
    
    return results

# Get index information
info = r.ft("documents").info()
print(f"Number of documents in index: {info['num_docs']}")