## The promise and the pitfalls of RAG
Retrieval Augmented Generation is the most common pattern for putting AI on top of enterprise data. The idea is simple: when a user asks a question, retrieve the most relevant documents from your own corpus and feed them to the model as context. The model then answers from your data instead of from its baked-in training set.
Done well, RAG dramatically reduces hallucination and lets you keep sensitive data inside your control. Done badly, it produces confident nonsense, surfaces documents the user should not see and quietly costs a fortune in tokens.
Getting it right is mostly about discipline in five areas: ingestion, retrieval, prompting, evaluation and access.
## Ingestion done right
The quality ceiling of any RAG system is set at ingestion. Garbage in, confident garbage out.
- **Parse properly.** PDFs, slide decks and scanned documents need real OCR and structure-aware parsers, not naive text extraction. Tables, headings and lists carry meaning that flat text loses.
- **Chunk with intent.** Split documents at natural boundaries such as sections and paragraphs, then add a small overlap. Avoid arbitrary character splits that cut sentences in half.
- **Enrich with metadata.** Capture the source, owner, sensitivity, last-updated date and any access tags. Every chunk should carry the metadata of its source.
- **Refresh.** Stale knowledge is dangerous. Set a refresh cadence per source and respect deletions in the source system.
## Retrieval that actually retrieves
Vector search alone is rarely enough. The patterns that win in 2026 combine several signals:
- **Hybrid search** that fuses dense vector similarity with classic keyword search. Vectors handle paraphrase, keywords handle exact terminology.
- **Reranking** with a small cross-encoder model on the top candidates. This single step routinely lifts answer quality by ten to thirty percent.
- **Filtering by metadata** before retrieval, so a salesperson never receives a chunk from an HR document by accident.
- **Query rewriting** for ambiguous or short questions. The model expands the query into a more retrievable form before search.
Measure retrieval in isolation. If the right document is not in the top five hits, no amount of clever prompting will save the answer.
## Prompting and grounding
The prompt that produces the answer should be boring and predictable. Include:
- A clear instruction to answer only from the supplied context.
- An explicit statement of what to do when the context does not contain the answer.
- The retrieved chunks, each with a citation marker.
- A required output format that includes citations.
Force the model to cite. If the user can verify the source in one click, hallucinations become obvious and trust grows.
## Evaluation as a habit
Build a golden set of question and expected answer pairs from real users. Run it on every change to the ingestion pipeline, the retriever, the model or the prompt. Track:
- **Retrieval accuracy.** Did the right chunk appear?
- **Answer faithfulness.** Did the answer stay within the retrieved context?
- **Answer correctness.** Did it actually answer the question?
- **Latency and cost** per query.
LLM-as-a-judge can score faithfulness automatically, but always sample manually as well. Human review catches drift that automated scores miss.
## Access control and privacy
Every retrieval must respect the same access policies as the underlying source. Two patterns dominate:
- **Filter at query time** using the user\u2019s identity, group memberships and clearance to restrict the candidate set.
- **Per-user indexes** for highly sensitive corpora, accepting the storage cost in exchange for hard isolation.
Never rely on the model to refuse to share retrieved content. The control must be in the retrieval layer.
## Observability and feedback
Log the question, the rewritten query, the retrieved chunks, the prompt, the answer and the user\u2019s reaction. This trace is gold for debugging, evaluation and continuous improvement. Add a simple thumbs-up/thumbs-down on every answer and route negatives into a review queue.
## A pragmatic build order
1. Ship a thin vertical slice on a small, well-curated corpus. 2. Add hybrid search and reranking once you can measure retrieval. 3. Add evaluation before you add features. 4. Layer access control and metadata filters before you broaden the corpus. 5. Expand to more sources only when the quality bar is met.
RAG is not a magic ingredient. It is a system, and like any system it rewards careful engineering with users who actually trust the answers.