All insights

Designing the Vector Index Behind Production Semantic Search

November 21, 2026 · 8 min read

Vector SearchEmbeddingsEngineering
Designing the Vector Index Behind Production Semantic Search

Semantic search demos are easy: embed a few documents, insert the vectors, and return the nearest neighbors. Production systems are harder because the vector index sits inside a distributed retrieval pipeline with changing data, uneven traffic, access controls, and strict latency targets.

The database is only one part of that pipeline. Embedding quality, chunk boundaries, metadata design, candidate retrieval, reranking, and update behavior all affect the result users see. Engineering leaders should evaluate the system as a whole rather than treating the vector store as a specialized replacement for a keyword index.

Start with a measurable retrieval contract

Before selecting a database or index algorithm, define what retrieval must accomplish. “Find semantically similar content” is not a useful production requirement.

A workable contract specifies:

  • The unit returned: passage, product, ticket, image, user, or document.
  • Corpus size today and expected growth over 12 to 24 months.
  • Query throughput, concurrency, and traffic peaks.
  • Latency targets for candidate retrieval and the complete request.
  • Freshness requirements for inserts, updates, and deletions.
  • Mandatory filters such as tenant, region, language, permissions, or status.
  • Relevance metrics such as recall@k, precision@k, or normalized discounted cumulative gain.
  • Cost limits per thousand queries and per million stored items.

This contract exposes trade-offs early. A support assistant searching ten million mostly static passages has a different architecture from a marketplace indexing rapidly changing inventory with hard geographic filters.

Build a representative evaluation set before tuning infrastructure. Include common queries, ambiguous terms, rare concepts, filters, and queries expected to return nothing. Human judgments are expensive, but even a few hundred reviewed query-result pairs provide a better baseline than anecdotal testing.

Treat embeddings as versioned application data

An embedding is a model-dependent representation, not a permanent property of a document. Changing the model, input prefix, normalization method, or chunking logic changes the vector space. Old and new vectors generally should not share the same index.

Store enough lineage to reproduce every vector:

  • Source identifier and source revision.
  • Embedding model and model version.
  • Preprocessing and chunking version.
  • Vector dimension and normalization method.
  • Creation time and pipeline run identifier.
  • Hash of the embedded text or payload.

Use an immutable embedding version as part of the collection or namespace design. When upgrading models, populate a parallel index, run offline comparisons, send a small percentage of traffic to it, and switch only after it meets the retrieval contract. Keep the previous version available for rollback until the new index has operated reliably.

Do not recompute an entire corpus for every source update. Hash the canonical embedding input and generate a new vector only when that input changes. This reduces model cost and shortens reindexing windows.

Choose chunks for retrieval, not storage convenience

Documents are often divided by a fixed token count because it is easy. That approach can separate a heading from its explanation, merge unrelated sections, or duplicate large amounts of text through overlap.

Prefer chunks aligned with structural boundaries: sections, paragraphs, product records, source-code symbols, or conversation turns. Keep the parent document identifier so retrieval can aggregate neighboring chunks or reconstruct broader context later.

Chunk size creates a real trade-off. Small chunks improve localization but can lose context and multiply index entries. Large chunks preserve context but dilute the signal and consume more downstream tokens. Test several strategies against labeled queries rather than choosing a universal number.

For long or hierarchical content, consider two-stage retrieval. Search concise section representations first, then retrieve passages within the selected sections. This can reduce the search space while preserving useful context.

Select the index from workload evidence

Exact nearest-neighbor search compares the query with every vector. It offers perfect recall but becomes expensive as the corpus grows. Approximate nearest-neighbor indexes trade some recall for lower latency and cost.

Graph-based indexes such as HNSW are strong general-purpose choices for low-latency retrieval. They often provide high recall and support incremental insertion, but their memory overhead can be substantial. Build parameters influence memory and construction time; query parameters control the latency-recall trade-off.

Inverted-file approaches partition vectors into clusters and probe only selected partitions. Combined with product quantization, they can compress very large collections. The savings come with more training, tuning, and possible recall loss. They are often attractive when vectors no longer fit economically in memory.

Disk-oriented ANN designs can support large corpora with lower RAM requirements, although storage latency and cache behavior become central capacity concerns.

Benchmark with your own vector distribution and filters. Vendor results based on random vectors, warm caches, or unfiltered queries are not enough. Measure recall and tail latency together. A fast index that misses relevant candidates simply transfers failure downstream.

Make filtering part of index design

Metadata filtering is where many vector-search designs fail at scale. A query may need similarity plus tenant isolation, document status, language, geography, product availability, and user permissions.

Post-filtering retrieves vector candidates first and removes invalid results afterward. It is simple but unreliable when filters are selective. If only one percent of the corpus is eligible, retrieving the nearest 100 vectors may produce no usable results.

Pre-filtering narrows the eligible set before vector search. It preserves policy boundaries but may reduce the efficiency or accuracy of an ANN index, depending on the database implementation. Some systems use filter-aware traversal or hybrid strategies to address this.

Test realistic combinations, especially high-cardinality tenant filters and permission predicates. Never depend on post-filtering alone for access control. Authorization constraints must be applied inside the trusted retrieval boundary, and cached results must include the security context in their key.

Avoid creating one physical index per small tenant unless isolation requirements demand it. Thousands of tiny indexes create operational overhead and poor resource utilization. Shared indexes with indexed tenant metadata are usually simpler, while large or regulated tenants may justify dedicated partitions.

Separate candidate generation from final ranking

Vector similarity should usually produce candidates, not the final order. Embeddings capture broad semantic relatedness but can underweight exact identifiers, dates, prices, negation, and domain-specific terms.

A robust query path may combine:

  1. Query normalization and intent-specific rewriting.
  2. Vector retrieval for semantic candidates.
  3. Lexical retrieval for exact terms and rare identifiers.
  4. Metadata and authorization constraints.
  5. Score fusion and deduplication.
  6. A reranker applied to a limited candidate set.

Hybrid retrieval is especially useful for enterprise corpora containing acronyms, error codes, contract numbers, and product names. Reciprocal rank fusion is a practical starting point because it combines ranked lists without assuming comparable score distributions.

Rerank only enough candidates to improve quality. Cross-encoders and LLM-based rankers can be accurate but add latency and cost. Set separate budgets for retrieval, reranking, and any subsequent generation step.

Engineer updates, deletions, and failure recovery

Production corpora change. Source records are corrected, permissions are revoked, products expire, and users request deletion. The indexing pipeline therefore needs database-grade operational behavior.

Use idempotent jobs and stable vector identifiers. Write source revisions to a durable change stream or queue, then let workers perform chunking, embedding, and indexing. Track progress per revision so retries do not create duplicates.

Plan explicitly for deletion. Tombstones may hide records quickly, but physical removal and index compaction can lag. Document that lag and verify it against privacy obligations. If backups contain vectors or embedded text, retention policies must cover them too.

Run reconciliation jobs that compare the source of truth with index manifests. Monitor missing vectors, orphaned vectors, stale revisions, duplicate identifiers, and embedding failures. A successful API response from the vector database does not prove the corpus is complete.

Capacity-plan from vectors and query behavior

Raw vector storage is easy to underestimate. One million 1,536-dimensional vectors stored as 32-bit floats require roughly 6.1 GB before metadata, graph edges, replicas, write-ahead logs, and temporary build space. A graph index and replication can multiply the practical footprint.

Estimate capacity with:

  • Vector count and dimensions.
  • Numeric precision or compression method.
  • Index overhead per vector.
  • Metadata and original payload size.
  • Replica count and failure-domain requirements.
  • Growth, reindexing, and compaction headroom.
  • Cache hit rate and storage throughput.

Load tests should include concurrent inserts, mixed filters, cold-cache queries, node loss, and index rebuilds. Report p95 and p99 latency, not averages. Also track the fraction of the latency budget consumed before reranking or generation begins.

Instrument retrieval as its own service

Expose metrics for embedding throughput, queue age, index freshness, candidate counts, filter selectivity, empty-result rate, ANN latency, reranker latency, and recall on a fixed evaluation suite. Segment them by index version, tenant class, language, and query type.

Log identifiers and scores needed for debugging, but avoid recording sensitive query or document text by default. A retrieval trace should explain which retrievers ran, which filters applied, how lists were fused, and why a result was removed.

Relevance regressions often arrive through ordinary changes: a new embedding model, a chunking adjustment, a metadata migration, or an index-parameter update. Put retrieval evaluations in the release path and use shadow traffic for major changes.

Short takeaway

Build semantic search around a retrieval contract, version embeddings, test filters with realistic selectivity, and benchmark recall against tail latency. The winning vector database is the one that keeps those guarantees during updates, failures, and growth—not the one with the fastest isolated demo.