Scaling RAG Architecture with Milvus Vector Databases in Production

Beyond the Prototype: Vector Database Bottlenecks

Building a Retrieval-Augmented Generation (RAG) prototype with a few thousand documents using an in-memory store like FAISS works fine. However, when moving to production handling tens of millions of dense embeddings, concurrent read/write queries, and requiring high availability in-memory stores launch out of RAM and fall short on throughput.

Scaling a RAG setup requires a distributed vector database like Milvus. The primary technical challenge is selecting the appropriate index type and tuning search parameters to achieve the optimal balance between recall accuracy and query latency.

Configuring High-Performance HNSW Indexes in Milvus

To scale vector search, you need a collection schema that supports metadata filtering alongside an index tuned for your target throughput. The example below uses the pymilvus SDK to define a collection with a Hierarchical Navigable Small World (HNSW) index and run a partitioned hybrid search.

from pymilvus import Collection, FieldSchema, CollectionSchema, DataType, connections
# Establish gRPC connection to the Milvus cluster
connections.connect("default", host="milvus-cluster.default.svc.cluster.local", port="19530")
# Define schema with scalar and vector fields
fields = [
 FieldSchema(name="doc_id", dtype=DataType.INT64, is_primary=True, auto_id=False),
 FieldSchema(name="department", dtype=DataType.VARCHAR, max_length=100),
 FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=1536) # e.g., OpenAI embeddings
]
schema = CollectionSchema(fields, description="Enterprise Knowledge Base", enable_dynamic_field=True)
collection = Collection("enterprise_rag", schema)
# Configure the HNSW Index
index_params = {
 "metric_type": "COSINE",
 "index_type": "HNSW",
 "params": {"M": 16, "efConstruction": 200}
}
collection.create_index(field_name="embedding", index_params=index_params)
collection.load() # Load collection into memory for querying
# Run a hybrid search with runtime EF tuning
search_params = {"metric_type": "COSINE", "params": {"ef": 64}}
results = collection.search(
 data=[query_vector],
 anns_field="embedding",
 param=search_params,
 limit=5,
 expr="department == 'engineering'", # Scalar pre-filtering
 output_fields=["doc_id", "department"]
)

Architectural Tuning Breakdown

• Schema Setup & Connection: We connect to the cluster and define a schema that includes a scalar field (department). Pure vector search often returns semantically close matches that fail business constraints. Adding scalar metadata allows you to filter vectors by exact category matches.
• HNSW Index Configuration: We choose HNSW over standard IVF (Inverted File). The parameter M=16 sets the max number of bi-directional links per node. A higher M improves recall, but increases memory overhead. efConstruction=200 controls search depth while building the index; setting this higher slows down ingestion but produces a graph structure optimized for fast reads.
• Hybrid Search & Query Tuning: When querying, we filter using expr="department == 'engineering'". Milvus evaluates scalar filters via bitsets before traversing the graph, limiting vector evaluation strictly to engineering documents. At runtime, ef=64 sets the graph search depth, allowing you to dial in the exact trade-off between recall accuracy and latency.

Properly tuning HNSW parameters and pre-filtering with scalar metadata keeps query latencies low and predictable, even as knowledge bases expand to millions of vectors.