Build RAG as a Search System, Not a Prompt Trick

Retrieval-augmented generation is often drawn as a simple loop: embed a question, retrieve documents, place them in a prompt, and ask a large language model for an answer.

That diagram is useful for a prototype. It is inadequate for production.

A reliable RAG application is closer to a search product with a probabilistic response layer. It needs query understanding, access control, ranking, evidence assembly, citation handling, observability, and clear behavior when the available knowledge is insufficient. Treating retrieval as a small preprocessing step produces systems that answer fluently while missing the right evidence.

The architectural priority is not maximizing how much context reaches the model. It is ensuring that the smallest defensible evidence set reaches the model, with provenance intact.

Separate the online and offline paths

A production design should have two independently operated paths.

The offline path prepares source material for retrieval. It ingests content, extracts structure, normalizes metadata, creates searchable units, builds indexes, and records lineage. The online path accepts a user request, resolves identity and permissions, retrieves candidates, ranks evidence, generates a response, and returns citations.

This separation matters operationally. Content ingestion can fail without taking down question answering. An index can be rebuilt or migrated while the serving path continues using the previous version. Retrieval releases can also be tested independently from prompt or model changes.

A practical offline path includes:

• Source connectors with incremental synchronization and deletion handling
• Parsing that preserves headings, tables, lists, and document hierarchy
• Stable source IDs and version IDs for traceability
• Metadata enrichment for ownership, dates, products, regions, and permissions
• Multiple indexes when lexical and semantic retrieval serve different needs
• Quality checks before new index versions become active

Do not let application code write arbitrary chunks directly into a vector database. Put an ingestion service between sources and indexes, and make index publication an explicit release operation.

Make retrieval a staged pipeline

One vector search is rarely enough. Enterprise questions contain product names, error codes, dates, acronyms, and implied filters. Dense retrieval may capture meaning but miss exact identifiers. Keyword search may find the identifier while ignoring conceptual relevance.

Use a staged retrieval pipeline instead:

1. Classify the request and extract constraints such as product, tenant, region, document type, and date.
2. Run lexical and semantic retrieval in parallel where appropriate.
3. Merge and deduplicate candidates using stable document identities.
4. Apply permission and freshness filters before content enters the generation context.
5. Rerank the reduced candidate set with a stronger model or deterministic business rules.
6. Assemble evidence while preserving source boundaries and citations.

Hybrid retrieval should be the default for mixed enterprise content. Exact-match signals are particularly important for source code, tickets, policy numbers, part numbers, and log messages.

Reranking is also one of the highest-leverage places to spend latency. Retrieving 50 inexpensive candidates and reranking 10 is usually better than sending the top results from a single approximate search directly to the LLM.

Keep authorization outside the model

Permissions are a retrieval concern, not a prompt instruction.

Resolve the caller's identity before retrieval and convert it into enforceable filters. If the underlying search engine cannot reliably filter by access-control attributes, partition indexes by security boundary or introduce an authorization-aware retrieval service. Do not retrieve prohibited content and ask the model not to mention it.

Authorization must apply to every route through the architecture, including query expansion, caches, rerankers, conversation history, logs, and citation previews. A generated answer may be safe while a displayed source snippet leaks restricted information.

Preserve the authorization decision with the response trace. When an employee asks why a document was not considered, operators should be able to distinguish low relevance from denied access.

Build context as structured evidence

Context assembly should not be string concatenation. Give the generation layer structured evidence containing source ID, title, section, version, timestamp, and text. The prompt renderer can transform that structure into the model-specific format.

This indirection has three benefits. It keeps provenance available after generation, allows models to be changed without rewriting retrieval, and makes context selection observable.

Set explicit context budgets by purpose. Reserve tokens for instructions, the user's request, evidence, conversation state, and the answer. When evidence exceeds its budget, remove low-value items or compress them with an auditable process. Do not silently truncate the final prompt at an arbitrary token limit.

Avoid mixing retrieved facts with untrusted instructions from source documents. Retrieved content should be clearly delimited and described as evidence, not executable direction. Documents can contain malicious text, stale operational commands, or examples that resemble system instructions.

Design for no-answer behavior

Every RAG system needs an abstention path. Without one, the LLM will often bridge evidence gaps with plausible general knowledge.

Abstention should depend on more than a single similarity threshold. Useful signals include poor ranking scores, disagreement between retrieval methods, missing required entities, weak citation coverage, and contradictions among sources. The application can then ask a clarifying question, state that no supported answer was found, or route the request to a human or another system.

Require generated claims to map to evidence when the use case demands grounded answers. Citation presence alone is not enough: a citation can be real but fail to support the sentence beside it. The response layer should verify that cited passages entail important claims, especially for legal, financial, medical, or operational guidance.

The user experience should communicate uncertainty plainly. “I found no current policy for Brazil” is more useful than a polished answer assembled from an outdated global document.

Observe the retrieval decisions

Standard application metrics such as uptime and total latency do not explain bad RAG answers. Capture a trace for each request with privacy-aware retention:

• Original and transformed queries
• Applied identity, filters, and security scope
• Candidate IDs and scores from each retriever
• Reranking results and exclusion reasons
• Evidence included in the final context
• Index, prompt, retriever, reranker, and model versions
• Citations returned and abstention decisions
• Latency and cost for each stage

This trace lets engineers determine whether a failure came from ingestion, retrieval, ranking, context assembly, or generation. Without it, teams tend to keep changing prompts because that is the only visible component.

Monitor retrieval health separately from answer quality. Watch empty-result rates, stale-source rates, permission-filter rejection rates, duplicate candidates, index lag, and the share of answers supported by current sources. Segment these metrics by content domain and request type; global averages hide weak collections.

Use replaceable architectural seams

Models, embedding services, search engines, and rerankers will change. Keep their interfaces narrow and versioned.

The retrieval service should return normalized evidence objects rather than vendor-specific search results. The generation service should consume those objects rather than querying an index directly. Source connectors should publish canonical documents rather than embeddings. Caches should include index and model versions in their keys.

These boundaries may look excessive for an initial release, but they prevent expensive coupling later. They also let teams compare retrieval strategies, roll back an index, or route workloads without rewriting the product surface.

Start with one deployable application if that suits the team. Logical boundaries do not require dozens of microservices. They require explicit contracts, ownership, and telemetry.

Short takeaway

RAG is not a shortcut around search engineering. Build separate ingestion and serving paths, combine retrieval methods, enforce permissions before generation, preserve evidence structure, and make abstention a first-class outcome. The LLM should explain the best authorized evidence available—not compensate for an architecture that failed to find it.