RAG Systems - Retrieval-Augmented Generation Architecture

Retrieval-Augmented Generation is the production pattern that lets a fixed-weight LLM answer questions over your private corpus without retraining. In practice, a RAG system is three pipelines stitched together: an ingestion pipeline that chunks and embeds documents, a retrieval pipeline that mixes vector similarity with lexical and metadata filters, and a generation pipeline that packs ranked context into the model window.

The hard parts are not the diagram, they are the defaults. Default chunking shreds tables. Default top-5 retrieval misses 30-50% of relevant passages. Default cosine search confuses synonyms with negations. This page walks through the architecture choices that move a demo RAG from 60% accuracy to a production-grade system at 90+%, with concrete numbers from Anthropic, Pinecone, Qdrant, Weaviate, LlamaIndex and Hugging Face evaluations.

What RAG Actually Is in Production

RAG is not a model, it is a search problem with a generative tail. The LLM is the cheapest, most replaceable component. Quality lives in retrieval, and retrieval quality is dominated by how you chunk, embed, filter, and rerank.

Retrieval first, generation second: top-k recall above 90% at k=20 is the gating metric
The corpus is the product: stale or duplicated documents poison answers more than weak prompts
Context window is not a substitute: stuffing 200K tokens hurts precision and latency vs ranked top-20
Grounding via citations is mandatory: every answer span should map to a retrieved chunk ID
Eval is continuous: every embedding swap, chunker change, or reranker tweak needs regression run
The cost driver is embedding inference and reranking, not generation, once you scale past a few million chunks

End-to-End Architecture (6 Stages)

Skipping any one stage caps quality at roughly 70% answer accuracy on real corpora.

Chunking: split source documents into 200-800 token units that preserve semantic boundaries
Embedding: encode each chunk with a dense model (768-3072 dimensions) and store the vector
Vector store: index with HNSW or IVF-PQ for sub-50ms ANN search at 10M+ vectors
Retrieval: fetch top-50 to top-150 candidates using hybrid (dense + BM25 + metadata filters)
Reranking: pass candidates through a cross-encoder to reorder, keeping top-5 to top-20
Generation: format ranked chunks with citations into the prompt and call the LLM

Chunking Strategies

Chunking is the single highest-leverage tuning knob. Hugging Face evaluations show chunk size alone can swing answer accuracy by 15-25 points on the same corpus.

Fixed-size: 256, 512, or 1024 tokens with 10-20% overlap. Fast, brittle on tables and code
Recursive character splitting: paragraph → sentence → word boundaries. Usual right starting point
Semantic chunking: split where embedding similarity between adjacent sentences drops below threshold
Document-structure-aware: respect markdown headings, HTML sections, PDF layout, code function boundaries
Decoupled chunks (LlamaIndex): embed a small summary but feed larger surrounding window to LLM at generation
Contextual chunking (Anthropic): prepend 50-100 token LLM-generated summary of how chunk fits in document. Reduces retrieval failures by 35%

Embedding Model Selection

Pick by domain fit and dimension budget, not leaderboard rank. Storage and query latency scale linearly with dimensions; recall does not.

OpenAI text-embedding-3-small: 1536 dims (truncatable to 512), strong general baseline, $0.02/1M tokens
OpenAI text-embedding-3-large: 3072 dims, best OpenAI quality, 6x cost of small. Truncate to 1024 for most use cases
Voyage voyage-3 and voyage-code-3: top of MTEB for English and code; Anthropic used Voyage as strongest tested
BAAI BGE (bge-large-en-v1.5, bge-m3): best open-weight option, 1024 dims, self-hostable, multilingual via bge-m3
Cohere embed-v3: native int8 and binary embeddings, 32x storage reduction at 95+% recall retention
Domain-specific: MedCPT for biomedical (255M PubMed pairs), specialized embeddings beat general by 10-20 points
Fine-tuning: 1,000-5,000 in-domain query-passage pairs typically yields 5-15 point recall@10 lift

Vector Stores: Which One When

No universal winner. Pick on operational model (managed vs self-hosted), scale ceiling, and whether you already run Postgres.

Pinecone: fully managed, namespaces for multi-tenancy, dense+sparse+BM25. $1,500-$3,000/month at 10M vectors and 200 QPS
Qdrant: fastest at scale (p99 ~12ms at 10M vectors), rich payload filtering, self-hosted or cloud
Weaviate: built-in hybrid search and reranker modules, multi-tenancy first-class. Strong for multi-modal
pgvector: matches dedicated DBs at 1M scale with HNSW; ceiling ~50M per node. Near-zero marginal cost. Use pgvectorscale past 10M
Elasticsearch / OpenSearch: pick when you already run it for logs and need lexical-first hybrid
Milvus / LanceDB: Milvus for 100M+ scale with sharding; LanceDB for embedded, file-based, columnar workflows

Hybrid Search and Reranking

Pure vector search loses to hybrid + reranking on every honest benchmark. Anthropic measured 67% reduction in retrieval failures going from naive embeddings to contextual embeddings + BM25 + reranking.

BM25 catches exact tokens vectors miss: error codes, SKUs, function names, legal citations like "Section 230(c)(1)"
Fusion via Reciprocal Rank Fusion (RRF) with k=60 is the boring, robust default for merging dense and sparse
Metadata filters (date, tenant, doc type, language) applied pre-ANN cut candidate space by 10-1000x with no recall cost
Cross-encoder rerankers (bge-reranker-v2-m3, Cohere Rerank 3): slow (10-50ms per pair) but precise; run on top-50 to top-150
ColBERT / ColBERTv2 late-interaction: best quality per latency for reranking, ships in RAGatouille
Autocut: drop candidates after a sharp similarity-score cliff, preventing irrelevant filler

Evaluation

Build a 200-500 question golden set early, generate synthetically with an LLM then filter with critique agents (groundedness, relevance, standalone >= 4/5), re-run on every change.

Retrieval: recall@k (target 90+ at k=20), MRR (target 0.7+), nDCG@10 for ranked relevance
Faithfulness/groundedness: does every claim trace to a retrieved chunk? LLM-as-judge on Prometheus-style rubric
Answer relevance: does answer address question? Cosine similarity + LLM judge
Context precision: fraction of retrieved chunks actually used; low values = over-retrieval
Hallucination rate: target under 2% on factoid questions; measure with claim-level NLI against retrieved context
Tools: Ragas, TruLens, DeepEval, Phoenix, or homegrown GPT-4-class judge

Failure Modes and Advanced Patterns

"Lost in the middle": LLMs ignore middle of long contexts. Put highest-ranked chunks first and last, keep packed context under 8K tokens
Synonym / negation collapse: "not safe" and "safe" embed close. Use NLI-aware rerankers and explicit negation tests
Query rewriting: rewrite vague queries into 2-5 search-optimized variants, fan out, merge (RAG-Fusion)
HyDE (Hypothetical Document Embeddings): LLM drafts fake answer, embed it, search with that vector. Boosts recall on zero-shot domains
Multi-hop and GraphRAG: build entity-relation graph at ingestion; traverse at query time. Multi-hop: 86% vs 32% vector RAG
Contextual retrieval (Anthropic): contextual embeddings + BM25 + reranking for 67% reduction in top-20 retrieval failures at ~$1.02/M document tokens with prompt caching
Freshness: schedule re-ingestion, detect drift by tracking recall@k on held-out set per week, version your index

FAQ

What chunk size should I start with?

512 tokens with 50-100 token overlap, recursive character splitting on paragraph then sentence boundaries. Re-evaluate at 256 and 1024 once you have a golden eval set; chunk size routinely swings accuracy by 15-25 points.

Do I need a dedicated vector database?

Below 1M vectors, pgvector with HNSW on existing Postgres is almost always right. Between 1M and 50M, Qdrant or Pinecone win on latency and operational simplicity. Above 50M, dedicated stores or pgvectorscale become necessary.

Is reranking worth the latency?

Almost always yes. Cross-encoder rerank over top-50 candidates adds 100-300ms but lifts answer accuracy 10-20 points. Cohere Rerank 3 or bge-reranker-v2-m3 are standard. ColBERTv2 wins on quality per millisecond if you can self-host a GPU.

When should I use GraphRAG instead of vector RAG?

When questions require connecting info across documents or aggregations chunks cannot answer in isolation: "which customers in region X bought product Y after event Z". GraphRAG hits ~86% vs ~32% for vector RAG on multi-hop. For "find documents about X", vector RAG is faster and equally accurate.

How do I evaluate RAG without human annotators?

Generate synthetic eval set: sample 200-500 chunks, have a strong LLM write a factoid Q+A per chunk, run three critique LLMs (groundedness, relevance, standalone) and keep only items scoring 4+ on all three. Score answers with GPT-4-class LLM-as-judge.

How big a difference does the embedding model make?

On general English corpora, top models cluster within 3-5 points on MTEB. On specialized domains (medical, legal, code), domain-tuned or 2000-pair fine-tune beats general by 10-20 points recall@10. Pick dimensions on storage budget; 1024 is the modern sweet spot.

Cheapest way to dramatically improve a working RAG?

Add hybrid search (dense + BM25 via RRF) and a reranker. Per Anthropic measurements, reduces retrieval failures 50-67% over naive vector search, costs no model retraining, adds only hundreds of milliseconds.

Next step

Your situation isn't generic.
Neither should the conversation be.

A short call to map what rag systems looks like for your team. No obligation, no pitch, just clarity.

Fix my RAG pipeline See Agent Development

Senior architect · 16+ years shipping · Direct, no agency layers

RAG Systems