RAG Hallucinations: The Fixes That Actually Worked

TL;DR

• Hallucinations in a RAG system are almost always a retrieval problem first. Fix retrieval before touching the system prompt.
• Corpus hygiene — removing stale, duplicate, and contradictory documents — is the highest-leverage intervention and costs nothing in inference.
• Switching from pure vector search to hybrid BM25 + dense retrieval with RRF fusion consistently improves recall@10 and reduces hallucination on exact-match queries.
• A cross-encoder reranker added after hybrid retrieval is the single biggest precision gain most pipelines can make, but watch the 512-token truncation trap.
• Prompt-side fixes (citation instructions, refusal framing) are real but marginal — they smooth over retrieval problems rather than solving them.

---

The pattern is always the same. A client ships a RAG chatbot, users file bug reports, and the engineering response is to make the system prompt longer. "Only answer from the provided context. Do not make anything up. If you are unsure, say so." The hallucinations get slightly less frequent. The bugs keep coming.

The prompt is not the problem. Retrieval quality is the ceiling. You cannot generate your way out of a bad retrieval. Prompt instructions only run after the model has seen whatever the retriever handed it. If the retriever handed it three irrelevant chunks and missed the one that mattered, the best-written prompt in the world is working against bad inputs.

What follows is ordered by effect size, based on what research and production benchmarks show. Where I cite a specific number, I am citing a source, not a personal measurement.

What is a RAG hallucination, exactly?

A RAG hallucination is an answer a retrieval system's own sources do not support. That definition is sharper than it looks. It rules out factual errors from correct retrieval of wrong documents (a corpus problem) and rules in confident-sounding answers that the retrieved passages technically contradict.

A RAG-powered legal assistant returns "Section 4.2 of the contract requires written notice 30 days before termination" and the user is satisfied. The trace shows the retriever returned five chunks, none of which mention Section 4.2. The model invented the citation. End-to-end evaluation scored it correct. Retrieval-layer evaluation would have flagged it immediately — which is why you need to measure both layers separately.

Large models handle sufficient context well but, when context is insufficient, tend to answer anyway rather than abstain. Smaller models hallucinate even when the context was sufficient. For large models, the priority is getting good context in. For smaller models, you also need refusal calibration at the generation layer. Retrieval still comes first in both cases.

Fix 1: Corpus hygiene (before you touch any code)

Retrieval augmented generation fails when knowledge bases contain duplicate versions, deprecated content, or draft documents, as the system treats all data as equally valid and blends conflicting information into confident but wrong answers.

This is the fix most teams skip because it is not technically interesting. Common fixes like better chunking, hybrid search, or re-ranking are costly and insufficient if the underlying documents lack metadata about version, status, audience, or scope. Tag every document with a status field (active, deprecated, draft) and a valid_until date. Stale documents, low-confidence chunks, and access-restricted content need to be filtered before they can contaminate the response. Filter on status=active at query time, before semantic scoring runs.

Fix 2: Chunking strategy

Chunking is where most RAG pipelines silently fail. Fixed-size chunking, which splits every document into equal-length blocks of tokens, is the default in every tutorial and the wrong answer for most production workloads. It ignores document structure and creates chunks that split paragraphs, sentences, and ideas mid-thought.

Semantic chunking uses embedding similarity to detect topic boundaries. When the cosine similarity between consecutive sentences drops below a threshold, a new chunk begins. That keeps semantically complete ideas in the same chunk. Always include metadata with each chunk: source document, section heading, page number, and parent chunk ID. This enables citation, filtering, and hierarchical retrieval.

One constraint worth co-designing up front: most cross-encoders silently truncate at 512 tokens. If your chunks are longer than that, the reranker never sees the second half of each chunk. Your chunk size and your reranker token limit need to be decided together, not independently.

Fix 3: Hybrid retrieval (BM25 + dense + RRF)

Pure vector search has a structural blind spot that causes a specific hallucination class: dense vectors see a product SKU and find the closest embedding neighbor, which might be a different SKU — wrong answer, high confidence. A 2026 benchmark on financial documents with mixed text and tables confirmed that BM25 outperformed state-of-the-art dense retrieval, because financial documents are full of identifiers, numbers, and exact terminology that dense embeddings smooth over.

Dense-only retrieval hits 78% recall@10. Sparse-only BM25 lands at 65%. Hybrid search reaches 91% recall@10. That gap is the difference between a production-ready RAG system and one that hallucinates on edge cases. Hybrid retrieval fuses both via Reciprocal Rank Fusion — a rank-only algorithm that sidesteps the score-incompatibility problem that breaks naively-weighted pipelines.

If your RAG system uses pure vector search, adding BM25 is the single highest-impact retrieval upgrade you can make. Most major vector databases support hybrid search natively now. For notes on which storage layer fits which workload, see the pgvector vs Pinecone comparison.

Fix 4: Cross-encoder reranking

Hybrid retrieval gives you better recall. Reranking gives you better precision on the chunks that actually reach the model. Re-ranking is the single biggest precision gain you can add to any RAG pipeline. A reranker takes the initial retrieval results and reorders them using a more expensive but more accurate model. Cross-encoders deliver 10–25% additional precision on top of hybrid retrieval and measurably reduce hallucinations.

Use Cohere Rerank (4096 token limit) or Jina Reranker v3 (8192 token limit) if your chunks exceed 512 tokens. On cost: applying a reranker to millions of documents at query time is architecturally incorrect — the first-stage ANN retrieval exists precisely to make reranking tractable. Rerank the top-20 to top-50 candidates from retrieval, not the full corpus. Also account for context position: research on the "lost in the middle" effect found accuracy highest when the relevant passage sat near the beginning or end of the context window — so after reranking, place your highest-scored chunk first.

Fix 5: Measurement — you cannot improve what you cannot see

None of the above are improvements until you can measure them. RAGAS provides four key metrics: Faithfulness (does the answer stick to retrieved context?), Answer Relevancy (does the answer address the question?), Context Precision (are the retrieved documents actually relevant?), and Context Recall (did retrieval find all the relevant documents?).

Target scores for production: Faithfulness above 0.9, Answer Relevancy above 0.85, Context Precision above 0.8. If Context Precision is low, fix your retrieval. If Faithfulness is low, fix your prompt or add guardrails. The diagnostic direction — which layer is broken — matters as much as the scores themselves. You need trace-level logging across every pipeline stage — what was retrieved, scored, passed to the model, and generated. Without this, debugging a bad answer is impossible.

Where prompting actually helps

After all of the above, prompting still matters. Two moves hold up: instruct the model to cite the chunk it drew from (which surfaces faithfulness failures in your eval traces), and give it an explicit refusal path so that the model refuses appropriately when the corpus does not answer the question. Over-refusing is a UX problem; under-refusing is a safety problem. What prompting does not fix is a retriever that returned the wrong chunks. A model instructed to ground itself will hallucinate with good style when the context is bad.

Fix order by effect size

Fix	Effect size	Implementation cost	When to skip
Corpus hygiene	High — removes entire hallucination classes	Time only, no infra cost	Corpus is already well-governed with status metadata
Semantic chunking	High — preserves context coherence per chunk	Low — one-time pipeline change	Documents are already short and structurally uniform
Hybrid retrieval (BM25 + dense + RRF)	High — 78% to 91% recall@10 in published benchmarks	Low-medium — most vector DBs support it natively	Corpus has no exact-match or identifier queries (rare)
Cross-encoder reranking	Medium-high — 10–25% precision gain over hybrid alone	Medium — adds latency and API cost	Hard latency budget under ~200 ms end-to-end
Context position ordering	Medium — documented attention bias at middle positions	Near-zero — prompt assembly change only	Passing a single chunk to the model
Prompt-side citation and refusal	Low-medium — reduces generation-layer hallucination	Zero	Never skip — always do this after fixing retrieval

What this looks like on a real project

The Biz365 AI project involved building an AI layer on top of structured business data. The pattern that held there is that the quality of what you index determines the quality of what the model can say. Embedding clean, well-structured data with clear metadata boundaries is not glamorous work, but it determines whether your faithfulness score is 0.6 or 0.9.

If you are scoping a RAG build or debugging an existing one, the RAG and LLM development page covers how I approach retrieval architecture, corpus design, and evaluation setup. If you are still deciding whether RAG is the right pattern for your use case at all, the Fine-Tuning vs RAG decision table will help you make that call before committing to an index pipeline.

FAQ

My RAG system was working fine and then started hallucinating more — what changed?

The most common cause is corpus drift: new documents were added that contradict older ones, or documents were updated without the old versions being deprecated. Check what was added to the index around the time hallucinations started. A last_indexed timestamp on every chunk makes this diagnosis much faster.

Does switching to a larger LLM fix RAG hallucinations?

Sometimes, partially. Larger models are better at abstaining when context is genuinely missing. But as ICLR 2025 research found, even large models tend to answer rather than abstain when context is insufficient — they just hallucinate more fluently. Fix retrieval first, then evaluate whether a model upgrade moves the remaining metrics.

How do I know if my hallucination is a retrieval problem or a generation problem?

Run RAGAS and look at Context Precision separately from Faithfulness. Low Context Precision means your retriever is returning the wrong chunks. High Context Precision but low Faithfulness means the model is ignoring correct context — and prompt-side fixes apply there. Most teams that skip per-layer evaluation cannot answer this question and apply the wrong fix.

What evaluation tooling should I set up before making any of these changes?

At minimum: RAGAS for per-layer metrics, and trace-level logging that captures what chunks were retrieved and scored for every query. Without the logging you cannot tell whether a fix helped in production versus only in your eval set. LangSmith and Arize are the two tools I see used most in production RAG observability setups.