Why Your RAG System Returns Wrong Answers

Your RAG system gives wrong or hallucinated answers because the retrieval layer is returning bad context, and the model is faithfully answering from that bad context. The model is not the problem in 80 percent of production RAG failures I diagnose. Fix retrieval first.

I am Mahmoud Zalt, an independent senior AI systems architect with 16 years building production software. I founded Sista AI, where retrieval feeds a workforce of autonomous agents that has been live in production for the past year, so I have watched RAG fail in every way it can. I work with engineering teams as an AI architecture advisor to diagnose and rebuild RAG pipelines that are underperforming in production. This article gives you the concrete diagnostic framework I use.

The Anatomy of a RAG Failure

A standard RAG pipeline has five layers where failure can originate: ingestion (chunking and indexing), embedding (turning text into vectors), retrieval (semantic search, keyword search, or both), reranking (sorting results by relevance), and generation (the LLM producing an answer). Most teams I work with have spent weeks prompting the LLM and almost no time measuring retrieval quality. That is backwards.

The failure modes I see most often, in order of frequency:

• Chunks are too large or too small. A 2,000-token chunk buries the relevant sentence. A 50-token chunk strips it of context. The model gets either noise or fragments.
• The embedding model does not match the domain. A general-purpose embedding model produces poor similarity scores for legal, medical, or technical content with specialized vocabulary.
• Pure semantic search misses exact terms. A query for 'CVE-2024-4171' or 'Section 12(b)(2)(A)' has no useful semantic neighborhood. Cosine similarity on embeddings will retrieve unrelated documents.
• No reranking step. Top-k retrieval by cosine distance is a rough filter, not a relevance ranking. Without a cross-encoder reranker, mediocre chunks rank alongside good ones.
• Missing or ignored metadata. Retrieving the right text from the wrong document version, the wrong tenant, or outside the relevant date range is still a retrieval failure even if the semantic score looks good.
• Context window stuffing. Cramming 20 retrieved chunks into the prompt means the model has to do its own retrieval inside the context window, which it does poorly.

Chunking Strategy: The Most Underrated Decision

Chunking is where most RAG pipelines go wrong first. The default of 'split every 512 tokens with 50-token overlap' is not a strategy; it is a placeholder. Here is how I approach it in production.

Match chunk size to query type

Short factual queries (lookups, definitions, specific values) need small, focused chunks: 150 to 300 tokens. Reasoning queries (comparisons, summaries, multi-step analysis) need larger chunks with more surrounding context: 500 to 800 tokens. If your use case has both, use hierarchical chunking: index small child chunks for retrieval but pass the parent chunk to the model for generation.

Preserve semantic boundaries

Split on paragraph, section, or sentence boundaries, not on raw token counts. A mid-sentence split at token 512 destroys the meaning of both halves. Libraries like LangChain's RecursiveCharacterTextSplitter walk a hierarchy of separators (double newline, single newline, period, space) and produce much cleaner splits than a fixed-width slice.

Worked example: financial document RAG

A team I worked with had a fund analysis chatbot returning wrong figures. Their chunks were 1,024 tokens with fixed splits. A single chunk would contain the tail of one table and the header of another, and the model would conflate the two. The fix: split on table and section boundaries first, then chunk within sections at 400 tokens. Retrieval precision went from 61 percent to 84 percent on their test set before any other change.

Chunking for structured data

If your source is a database or structured JSON, do not serialize the whole row into one chunk. Flatten to sentence-level facts: 'Product X has a lead time of 14 days in region EMEA as of Q1 2025.' This makes the embedding semantically precise and the retrieved fact directly usable by the model without parsing.

Embedding Model Selection and Domain Fit

Not all embedding models are equal for your domain. text-embedding-3-small and text-embedding-ada-002 from OpenAI are solid general-purpose baselines. But if your corpus is dense with domain terminology, run a quick benchmark before committing.

How to benchmark embeddings in two hours

Build a small golden evaluation set: 50 to 100 (query, expected document) pairs drawn from real user questions and your actual corpus. Run retrieval with each candidate embedding model. Measure Recall@5 and Recall@10: what fraction of the time is the correct document in the top 5 or top 10 results? A model with Recall@5 of 0.72 versus 0.58 on your domain is a clear win regardless of MTEB leaderboard rankings.

Models worth benchmarking for domain-specific work: voyage-large-2-instruct (strong on technical/code), bge-large-en-v1.5 (good open-source baseline), Cohere embed-v3 (strong multilingually). Do not assume the most expensive general model wins on your specific data.

Embedding freshness and drift

If your corpus updates frequently, index new documents with the same model you used to embed the originals. Switching embedding models without reindexing the entire corpus causes cosine distance to be meaningless: you are comparing vectors from different geometric spaces. This is a silent failure that produces very confident but wrong retrievals.

Hybrid Search: Combining Semantic and Keyword Retrieval

Semantic search alone fails on precise identifiers, product codes, legal citations, and rare proper nouns. BM25 (keyword search) alone fails on paraphrase and conceptual queries. Hybrid search, running both and merging the results, handles both cases and is almost always worth the added complexity in production.

Reciprocal Rank Fusion

The cleanest merge strategy is Reciprocal Rank Fusion (RRF). For each document, compute a combined score as the sum of 1 / (k + rank_semantic) and 1 / (k + rank_bm25), where k is typically 60. Documents that rank well in both lists bubble up. Documents that rank highly in only one list still surface if the other list does not strongly contradict them. RRF requires no weight tuning and is robust across query types, which is why Elasticsearch, Weaviate, and Pinecone have all added native support for it.

When to weight one side higher

For a customer support bot over a knowledge base with consistent terminology, lean toward BM25 (weight 0.7 keyword, 0.3 semantic). For a research assistant over scientific papers where users ask conceptual questions, lean toward semantic (0.3 keyword, 0.7 semantic). Tune these weights using your eval set, not intuition.

Reranking: The Step Most Teams Skip

Bi-encoder embeddings (the kind used in vector databases) are fast but imprecise. They produce an approximation of relevance good enough for candidate selection. A cross-encoder reranker sees the full (query, document) pair together and produces a much more accurate relevance score, at the cost of latency. The pattern: retrieve top 20 to 50 candidates from hybrid search, rerank with a cross-encoder, pass only the top 5 to the LLM.

Reranker options

The latency cost is real but almost always worth it. In my experience, adding a reranker to a naive top-5 retrieval pipeline improves answer accuracy by 15 to 25 percent on factual queries, with no change to the LLM or prompt.

What teams get wrong with reranking

The most common mistake: reranking only 5 candidates instead of 20 to 50. If the correct document is not in the candidate set, reranking cannot save you. Retrieve wide, then rerank to narrow. Retrieve narrow and rerank, and you just added latency without benefit.

Metadata Filtering and Retrieval Guardrails

A retrieval system without metadata filtering is a liability in multi-tenant, versioned, or access-controlled applications. Semantic similarity does not enforce that a user only sees their own tenant's data, only retrieves from documents they have permission to access, or only gets answers from the current version of a policy document.

Filter before or after retrieval

Pre-filtering (filtering in the vector database query itself) is safer and faster: it limits the candidate set before cosine distance is computed. Post-filtering (filtering after retrieval) can silently return fewer than k results if many candidates are filtered out, which breaks the assumption that the reranker sees enough candidates. Use pre-filtering for hard constraints (tenant ID, document status, permission level) and post-filtering only for soft preferences.

Metadata schema design

Every chunk I index includes at minimum: source_id, tenant_id, created_at, doc_version, section_type, and access_level. The query layer always injects tenant_id and access_level filters from the authenticated session, never from user input. Injecting filter values from user-supplied query parameters is a data-isolation vulnerability.

Human-in-the-loop for low-confidence retrievals

When the top reranked score is below a threshold (tune empirically, typically 0.35 to 0.45 on a 0-1 scale), I surface a 'I could not find a confident answer' response rather than hallucinating from weak context. This is a guardrail, not a failure. Users trust a system that admits uncertainty far more than one that confidently gives wrong answers. Wire this to your observability stack and review low-confidence queries weekly.

Building a Measurable Retrieval Eval Loop

You cannot improve what you do not measure. Here is the eval loop I run for every RAG system I audit.

The four metrics that matter

• Recall@k: is the correct document in the top k results? Measures retrieval coverage. Target: Recall@5 above 0.80 for production.
• MRR (Mean Reciprocal Rank): how high does the correct document rank on average? Penalizes systems that find the right answer but bury it.
• Context Precision: of the chunks passed to the LLM, what fraction are actually relevant? Low precision means the model is working with noise.
• Answer Faithfulness: does the generated answer stay within the retrieved context, or does it add facts not in the chunks? This is the hallucination signal. Use frameworks like RAGAS or DeepEval to automate this with an LLM-as-judge approach.

Building the golden eval set

Start with 100 real questions from actual users or domain experts, each paired with the source document that should be retrieved and an expected answer. Synthetic eval sets generated by LLMs are useful to bootstrap but tend to be too easy; they miss the adversarial and ambiguous queries that cause production failures. Grow the set by logging low-confidence and negative-feedback queries from production, then labeling them.

The improvement loop

Run evals on every chunking or retrieval change before deploying. The sequence I use: change one variable (chunk size, embedding model, retrieval strategy, reranker) per experiment, measure the four metrics against the golden set, deploy only if Recall@5 and Faithfulness both hold steady or improve. One change at a time makes attribution clean. Changing chunking, embedding, and reranker simultaneously makes it impossible to know what helped.

Frequently Asked Questions

Why does my RAG chatbot give confident wrong answers?

Confident wrong answers almost always mean the model received a retrieved chunk that is plausibly relevant but factually wrong for the query, and the model treated it as ground truth. Check Context Precision: are the chunks you are passing actually about what the user asked? The fix is usually reranking, smaller chunks, or adding a confidence-gating guardrail that declines to answer when retrieval scores are low.

Does switching to a better LLM fix RAG hallucinations?

Rarely. A better model will hallucinate less when given good context, but it will still produce wrong answers when given bad context. I have seen GPT-4o produce confidently wrong answers when the retrieval layer was broken, and I have seen smaller models perform accurately when the chunks were clean and relevant. Fix the retrieval pipeline first; upgrade the model after you have measured a ceiling.

How many chunks should I pass to the LLM context?

Three to five well-reranked chunks is almost always better than ten to twenty loosely ranked ones. Beyond five chunks, the model tends to either average across conflicting information or anchor on whichever chunk appears first in the prompt (primacy bias). Retrieve wide (top 20 to 50 candidates), rerank aggressively, pass only the top 3 to 5. If your query type genuinely requires synthesis across many sources, consider a map-reduce pattern where each chunk is answered independently and the results are merged.

What is the fastest way to improve an existing RAG system?

Add a reranker and build an eval set, in that order. A reranker requires no re-ingestion and no change to your vector database schema. You can deploy Cohere Rerank or a self-hosted bge-reranker in a day and see measurable improvement immediately. The eval set makes every subsequent improvement provable rather than anecdotal. Without it you are making changes blind.

How do I handle RAG over documents that update frequently?

Implement incremental indexing: track a last_indexed_at timestamp per document and re-chunk and re-embed only documents modified since the last run. Never mix embeddings from different models in the same index. If you change embedding models, reindex the entire corpus. Use metadata fields like doc_version and valid_until to pre-filter queries so users never retrieve from stale superseded documents.

Should I use an off-the-shelf RAG framework or build custom?

Use a framework (LangChain, LlamaIndex, Haystack) to prototype and learn the problem space, then replace the components that do not fit your constraints with custom implementations. Frameworks make the first 70 percent fast and the last 30 percent painful. In production I commonly replace framework defaults for chunking, retrieval fusion, and reranking because the defaults are designed for the average use case, not yours.

Ready to Fix Your RAG Pipeline?

A RAG system that returns wrong answers erodes user trust faster than any other AI failure mode. The good news is that the failure is almost always in the retrieval layer, and retrieval failures are diagnosable and fixable with a structured approach: measure Recall@5 and Faithfulness first, then work through chunking, hybrid search, reranking, and metadata filtering one variable at a time.

If you need an experienced eye on your RAG architecture, or you are building a new system and want to avoid these failure modes from the start, I work with teams as an independent AI architecture advisor. You can read more about my background on my about page or see what I have shipped on my projects page. When you are ready to talk, reach out directly.

Work with me to fix your RAG architecture