Scale Semantic Search Without Losing Relevance
November 19, 2026 · 7 min read

Semantic search demos are easy. Embed a few thousand documents, place the vectors in an index, and return the nearest neighbors to a query. The results often look impressive.
Production systems are less forgiving. At scale, freshness, tenant isolation, metadata filtering, tail latency, and index cost matter as much as similarity. Relevance also degrades in ways that infrastructure dashboards cannot detect.
The core engineering lesson is simple: a vector database is one component of a retrieval system. It is not the system itself. Teams should design the complete path from content segmentation through ranking and evaluation before committing to a vendor or index type.
Start With the Retrieval Contract
Define what a successful search must return before choosing an embedding model. “Find semantically similar text” is not an operational requirement.
A useful retrieval contract specifies:
- The unit returned: passage, document, product, ticket, or entity
- Required filters, such as tenant, permissions, language, date, or region
- Freshness expectations for new, changed, and deleted content
- Latency targets at both median and tail percentiles
- Recall requirements for critical query classes
- How results will be consumed by people, APIs, or language models
These constraints change the architecture. An internal knowledge assistant may tolerate 500 milliseconds of retrieval latency but cannot leak one employee’s documents to another. Product discovery may need aggressive filtering and predictable performance during traffic spikes. Duplicate detection may prioritize recall over result ordering.
Write representative queries and expected results early. They become the foundation for evaluation and prevent infrastructure decisions from substituting for product decisions.
Treat Embeddings as Versioned Application Data
An embedding is a model-dependent representation, not a permanent property of a document. Changing the model, input formatting, segmentation logic, or vector dimensions creates a new data version.
Store enough information to reproduce and replace every vector:
- Source identifier and source revision
- Chunk boundaries and normalized text hash
- Embedding model and version
- Dimensions and distance metric
- Generation timestamp
- Access-control and business metadata
Do not overwrite an existing vector set during migration. Build the new version alongside it, validate coverage, compare relevance, and shift traffic gradually. Dual-read experiments are especially useful because they expose regressions in real query traffic without making rollback difficult.
Model upgrades should be justified by measured retrieval gains. A larger embedding model may improve a public benchmark while performing worse on part numbers, acronyms, or multilingual terminology specific to your domain. It can also increase storage, network traffic, and indexing time.
Chunk for the Unit of Use
Poor segmentation cannot be repaired by a better vector database. Chunks that are too large combine unrelated concepts and dilute similarity. Chunks that are too small lose context and create noisy, repetitive results.
Choose boundaries that match how information is consumed. Documentation often benefits from section-aware chunks that retain headings. Support cases may be represented by the issue, resolution, and product metadata rather than arbitrary token windows. Catalog search usually embeds structured product fields instead of slicing descriptions.
Overlap is not free. It increases vector count, storage, ingestion work, and duplicate results. Use it only when meaning routinely crosses boundaries, and collapse neighboring hits before presenting them downstream.
Keep the parent-child relationship explicit. Search can operate on smaller passages while returning a larger document or section. This separation often improves both recall and user experience.
Choose an Index From Workload Measurements
Exact nearest-neighbor search provides a useful relevance baseline but becomes expensive as the collection grows. Approximate nearest-neighbor indexes trade some recall for lower latency and cost.
Graph-based indexes such as HNSW commonly provide strong recall and fast reads, but consume significant memory and can make high-churn workloads operationally awkward. Inverted-file approaches partition the vector space and search selected partitions, offering tunable cost at the expense of more calibration. Quantization reduces memory by compressing vectors but may reduce accuracy.
There is no universally best configuration. Benchmark with your vectors, filters, update rate, and concurrency. Measure:
- Recall against an exact-search baseline
- p50, p95, and p99 query latency
- Performance under selective metadata filters
- Index build and update duration
- Memory, storage, and replica requirements
- Deletion behavior and compaction overhead
A million static vectors and a hundred million frequently updated vectors are different systems. So are a global corpus and thousands of small tenant-specific collections. Avoid extrapolating from vendor benchmarks that omit your filtering and mutation patterns.
Filtering Is Part of Search Correctness
Metadata filters are not merely an optimization. They enforce product scope, permissions, and data boundaries.
Applying filters after vector retrieval can fail when most nearest neighbors are ineligible. Requesting 20 candidates and discarding 18 unauthorized results leaves poor recall even if relevant authorized content exists deeper in the index. Over-fetching helps only until filters become highly selective.
Prefer systems that can incorporate filters during candidate generation, then test their behavior with realistic metadata distributions. Some index designs degrade when a query targets a small subset of a large collection.
Security filters require stricter treatment. Resolve authorization from trusted identity and policy data, not user-provided query text. Test denial cases explicitly. Where isolation requirements are high, separate indexes or namespaces may be safer than complex shared filters, even if they increase operational overhead.
Use Hybrid Retrieval and Reranking
Embeddings are strong at conceptual similarity but weak at some exact signals. Error codes, legal citations, names, SKUs, and newly introduced terminology often favor lexical search.
Production search should usually combine semantic and lexical candidates. Reciprocal rank fusion is a practical starting point because it combines rankings without requiring score calibration across different retrieval methods.
Then rerank a limited candidate set using stronger signals. These may include a cross-encoder, recency, authority, document quality, popularity, or business rules. Keep candidate generation broad and inexpensive; reserve costly ranking for tens of results, not millions.
A typical pipeline is:
- Normalize the query and determine its scope.
- Retrieve semantic and lexical candidates in parallel.
- Apply mandatory permissions and metadata constraints.
- Merge and deduplicate candidates.
- Rerank the top set using richer features.
- Return results with traceable scores and source identifiers.
This layered design is easier to tune than expecting one vector similarity score to represent every notion of relevance.
Engineer Ingestion for Freshness and Recovery
At scale, ingestion becomes a distributed data problem. Sources change, embedding APIs throttle, events arrive out of order, and deletions are missed.
Make each stage idempotent. Use stable identifiers derived from the source, revision, and chunk definition. Record processing state so failed work can resume without duplicating vectors. Separate extraction, chunking, embedding, and indexing queues to isolate bottlenecks.
Track freshness as measurable lag from source change to searchable state. Also run periodic reconciliation against source systems. Event-driven updates provide speed; reconciliation catches dropped events and silent drift.
Deletion deserves first-class handling. Removing content from the source is not enough if vectors remain queryable. Verify deletion across replicas, caches, and old index versions, and define how long tombstones must be retained.
Evaluate Relevance Before Optimizing Cost
Infrastructure metrics cannot tell you whether results are useful. Maintain a labeled query set covering common, difficult, and high-risk cases. Include short queries, exact identifiers, ambiguous language, filtered searches, and queries with no correct result.
Measure recall at candidate generation separately from final ranking quality. If the correct item never enters the candidate set, reranking cannot recover it. Useful measures include recall@k, mean reciprocal rank, normalized discounted cumulative gain, zero-result rate, and filter-correctness rate.
Online signals such as clicks or accepted answers can complement labels, but they are biased by result position and user behavior. Use controlled experiments rather than assuming engagement equals relevance.
Cost optimization should follow quality baselines. Common levers include reducing dimensions, quantizing vectors, moving cold partitions to cheaper storage, batching embeddings, and lowering replica counts. Re-run relevance and latency tests after every change.
Short Takeaway
Build semantic search as a layered retrieval service, not a vector-store integration. Version embeddings, benchmark indexes with real filters, combine semantic and lexical retrieval, and evaluate relevance continuously. Scale is manageable when correctness, freshness, and cost are explicit engineering targets.