RAG, Fine-Tuning, or Prompting: The Short Answer
Pick by the type of problem, not by what sounds most advanced. If your model needs current or private knowledge, use RAG. If it needs a new behavior or output format it cannot produce today, try prompting first. Fine-tuning earns its place only when prompting has plateaued and you have clean, labeled data at scale. That ordering covers roughly 90 percent of the AI feature decisions I see in production.
I am Mahmoud Zalt, an independent AI systems architect with 16 years building production software. I created Laradock (millions of Docker installs), built Apiato, and founded Sista AI. I now advise startups and engineering teams through my AI Architecture consulting practice. What follows is the decision framework I actually use, not a vendor pitch. Read more about me.
Why the Order of Your Question Already Reveals the Mistake
Teams usually arrive at this decision after seeing a demo or reading a blog post. They have a feature idea and they want to know which technique to deploy. That framing is backwards. The right question is: what is the gap between what the model does today and what you need it to do? The answer to that question directly maps to the technique.
- Knowledge gap: the model does not know your proprietary data, recent events, or internal documents. This is a retrieval problem. RAG closes it.
- Behavior gap: the model produces the right kind of content but not in the exact tone, structure, persona, or chain of reasoning you need. This is a prompting problem. Prompt engineering closes it.
- Capability gap: the model structurally cannot do the task, even with good prompts and relevant context. It might be confusing entity types, ignoring a constraint, or producing a consistently wrong schema. This is the narrow case where fine-tuning helps.
Notice: capability gaps are rare and expensive to confirm. The most common mistake I see is teams diagnosing a behavior gap, spending weeks preparing fine-tuning data, and then discovering a better system prompt would have solved it in an afternoon.
Prompting Is Not a Consolation Prize
Prompt engineering is systematically underestimated because it does not feel like engineering. It is. A well-structured prompt with a clear persona, explicit output format, worked examples (few-shot), and a chain-of-thought instruction can move accuracy by 20 to 40 percentage points on most tasks. I have seen teams spend three months and tens of thousands of dollars on fine-tuning to achieve a gain that a two-hour prompting session later matched.
What disciplined prompting actually looks like
Start with a system prompt that states the role, the constraints, the output format (schema or example), and a few representative worked examples. Use a temperature of 0 for deterministic tasks. Add an explicit instruction like 'Before answering, reason step by step in a scratchpad block' for reasoning-heavy tasks. Version your prompts in source control exactly like code. Evaluate every change with a fixed eval set of at least 50 representative inputs, not by eyeballing three outputs.
That last point is where most teams fail. Without a structured eval, you cannot tell if a prompt change made things better or just different. You end up in a loop of vibe-based tweaks. Build the eval harness first, then iterate.
When prompting genuinely maxes out
Prompting has a ceiling. You will hit it when: the base model lacks the domain vocabulary (technical jargon, acronyms, specialized notation); when the task requires consistent multi-step reasoning across many hops that exceeds reliable context use; or when latency from a long, detailed system prompt is a real product constraint. Only at this ceiling does fine-tuning become a serious candidate.
RAG: When Your Problem Is Knowledge, Not Capability
Retrieval-Augmented Generation solves one specific class of problem: the model does not have the information it needs at inference time. This covers internal knowledge bases, product documentation, recent news, customer records, legal corpora, codebase context, and anything that changes faster than you can retrain a model. If your problem is in this class, RAG is almost always the right first architecture.
A minimal production RAG stack
At its core: a document ingestion pipeline (chunk, embed, store in a vector database), a retrieval step at query time (embed the query, retrieve top-k chunks, rerank if needed), and a generation step where the retrieved context is injected into the prompt. The naive version is 50 lines of code. The production version handles chunk overlap, metadata filtering, hybrid search (dense plus sparse BM25), citation tracking, and staleness management.
Key numbers that actually matter in production: chunk size 256 to 512 tokens with 10 percent overlap works for most prose. Top-k of 5 to 10 with a reranker (cross-encoder or an LLM reranker) beats top-3 without reranking on precision. Embedding model quality matters more than vector DB choice for most teams under 10 million documents.
What teams get wrong with RAG
The most common failure mode is poor chunking. Teams split documents at fixed token counts, slicing mid-sentence or mid-table, and then wonder why retrieval quality is low. Chunk at semantic boundaries: paragraphs, sections, table rows. The second most common failure is skipping evaluation of the retrieval step independently of the generation step. If your retrieved chunks are wrong, no model will save you. Measure retrieval recall and precision on a labeled eval set before you touch the generation prompt.
The third failure: treating RAG as a fire-and-forget pipeline. Documents change. You need a re-ingestion strategy, a staleness detection mechanism, and an observability layer that lets you inspect what chunks were actually retrieved for any given query in production.
Fine-Tuning: The Narrow Legitimate Case
Fine-tuning is not a shortcut to a smarter model. It adjusts the model weights to reinforce a specific behavior distribution. It cannot inject knowledge reliably (that is RAG's job). It can teach the model to reliably produce a consistent schema, adopt a domain register, or execute a multi-step task it struggles with in prompting. But the bar for justifying it is high.
The checklist before you start a fine-tuning project
- You have at least 500 high-quality labeled examples, ideally 1000 to 5000. Less than that, and few-shot prompting usually wins.
- You have run a serious prompting experiment first. Not a one-hour attempt. A disciplined two-week effort with an eval harness.
- The capability gap is measurable. You have an eval showing the base model with best prompting scores X, and your target is Y. You know what 'done' looks like.
- You have a plan for ongoing maintenance. Fine-tuned models go stale. When the base model updates, you may need to re-run. When your task distribution shifts, your fine-tune may regress.
- You have budgeted for the full cycle: data labeling, training compute, validation, deployment, and monitoring. For a mid-size fine-tune on a frontier model, the real cost including engineering time is often five to fifteen times the raw training cost.
A concrete worked example where fine-tuning was right
A legal document classification task: a team needed to classify contract clauses into 40 proprietary categories that did not exist in any training corpus. The categories had subtle distinctions that could not be explained in a prompt short enough to be practical at scale. They labeled 3,000 examples with domain lawyers, fine-tuned a smaller model (not the frontier one), and achieved 94 percent accuracy versus 71 percent for best-prompt GPT-4. The smaller model also ran at one-tenth the cost per call. That is the legitimate fine-tuning story: specialized, high-volume, well-labeled, with a measured baseline. Notice they did not fine-tune the frontier model. They distilled the task into a cheaper specialized model, which is the economically rational outcome in most justified fine-tuning projects.
Treat Them as Composable Layers, Not Rivals
The most production-robust AI features I have built combine all three. The mental model is a stack: prompting is always present (it is how you talk to the model), RAG is injected when knowledge is needed (context window augmentation), and fine-tuning is a background optimization you apply to a downstream model when both of the above have been maximized. They do not compete. They address different layers of the same pipeline.
A concrete architecture that uses all three
Consider a customer-facing support assistant for a SaaS product. The base system prompt (prompting layer) defines the persona, tone, escalation behavior, and output format. At inference time, the user query triggers a retrieval pipeline (RAG layer) that pulls the relevant documentation sections and any open ticket context. The retrieved chunks plus the conversation history are injected into the prompt. Under the hood, the model serving this is a fine-tuned variant (fine-tuning layer) trained on 2,000 labeled examples of correct escalation decisions, because the base model was inconsistent on the escalation classification specifically. Each layer is independently tunable. You can improve retrieval quality without touching the fine-tune. You can refine the system prompt without re-ingesting documents. This separation of concerns is what makes the system maintainable.
The decision tree in four questions
- Does the model lack information it needs? Yes: add RAG. Then reassess.
- Is the output format, tone, or reasoning structure wrong? Yes: improve the prompt with examples and constraints. Measure with evals. Repeat.
- Is prompting plateaued on a measurable eval? Yes, with 500 plus labeled examples: consider fine-tuning a smaller, cheaper model for the specific sub-task.
- Are you combining these layers cleanly with observability on each? No: stop and add instrumentation before adding more complexity.
Evals and Observability: The Part Everyone Skips
None of the above decisions are durable without a measurement layer. Evals are not a nice-to-have. They are how you know if a change worked, how you prevent regressions, and how you justify the cost of fine-tuning to a skeptical stakeholder. Without them, you are doing aesthetics, not engineering.
Minimum viable eval setup
For most teams starting out: a golden set of 50 to 200 input and expected-output pairs, covering the distribution of real queries. An automated scorer (LLM-as-judge using a separate model and a rubric, or a deterministic scorer for structured outputs). A baseline run before any change. A delta report after. This is a day of engineering work and it pays back immediately. Tools like LangSmith, Braintrust, and PromptFoo make this faster but you can do it in a spreadsheet and a Python script to start.
Production observability for RAG specifically
Log: the raw query, the retrieved chunk IDs and scores, the final prompt sent to the model, and the response. This lets you diagnose retrieval failures (wrong chunks surfaced), prompt failures (right chunks, wrong synthesis), and model failures (hallucination despite correct context) separately. If you cannot distinguish these failure modes in production, you cannot improve your system systematically. I require this logging setup before any team I advise goes to production with a RAG feature.
Guardrails and cost controls
Set input and output token budgets explicitly. Use structured output schemas (JSON mode or tool-calling) wherever the output format is machine-consumed. Add a lightweight input classifier to catch off-topic queries before they hit your expensive retrieval and generation pipeline. These are not advanced optimizations. They are table stakes for a production AI feature that does not surprise you with a four-figure inference bill at the end of the month.
Security and Data Considerations Specific to Each Approach
The technique you choose changes your threat surface, and I want to name this explicitly because it is often left out of technique comparisons.
| Technique | Primary data risk | Key control |
|---|---|---|
| Prompting | Prompt injection via user input | Sanitize user-controlled input; never interpolate raw user text into privilege-bearing system prompt sections |
| RAG | Retrieval of documents the user is not authorized to see | Per-user or per-role metadata filtering at retrieval time, not post-retrieval |
| Fine-tuning | Training data memorization and exfiltration via extraction attacks | PII scrubbing before training data prep; differential privacy techniques for sensitive corpora; do not fine-tune on data you would not be comfortable the model reciting verbatim |
The RAG authorization failure is the one I see most often. Teams build a vector database, index all company documents, and then discover that customer A can retrieve chunks from customer B's documents because they forgot to scope retrieval by tenant. Add tenant ID as a required metadata filter on every retrieval query, not a post-filter on results.
Frequently Asked Questions
Should I use RAG or fine-tuning for my company knowledge base?
RAG, almost certainly. A knowledge base is by definition a knowledge problem, not a capability problem. The content changes, grows, and needs to be auditable. Fine-tuning knowledge into weights is expensive, produces a stale model the moment documents update, and gives you no citation trail. Build a RAG pipeline with good chunking, a reranker, and document-level access controls. Only consider fine-tuning if you also have a behavior problem in how the model uses that knowledge, and only after RAG is working well.
Is fine-tuning worth it for a custom tone or brand voice?
Usually not. Brand voice is almost always a prompting problem. A well-crafted system prompt with 5 to 10 annotated examples of on-brand responses, clear no-go phrases, and explicit tone adjectives will get you 80 to 90 percent of the way there in an afternoon. If you have high-volume production traffic and need to reduce token cost by moving to a smaller model, then fine-tuning that smaller model on your voice examples can make sense economically. But start with the prompt. Measure first.
How much labeled data do I actually need to fine-tune?
In practice, under 200 examples almost never justifies fine-tuning over few-shot prompting. 500 examples is a reasonable floor for a narrow, well-defined task. 1,000 to 5,000 is the range where you see reliable, measurable gains. Above 10,000 high-quality examples, you are in territory where a custom fine-tune can genuinely outperform frontier prompting for specialized tasks. Data quality matters more than quantity: 500 clean, consistent examples beat 5,000 noisy ones.
Can I combine RAG and fine-tuning in the same system?
Yes, and for complex production systems this is often the right architecture. Fine-tune a smaller model to handle the structured reasoning or classification sub-tasks reliably and cheaply. Use RAG to inject the dynamic, current, private context that fine-tuning cannot provide. Use a strong system prompt to tie the behavior together. Each layer is independently improvable and debuggable. The mistake is conflating the layers: do not try to inject knowledge via fine-tuning or teach behavior via retrieval.
What is the most common mistake teams make when choosing between these approaches?
Skipping evals and making the decision by feel. A team sees a few bad outputs from a prompted model and concludes they need fine-tuning. They spend two months on data prep and training. They deploy the fine-tuned model without a proper comparison. It feels better on the examples they remember, but they have no idea if it is actually better on the full distribution. The fix is always the same: build a 100-item eval set before you make any technique decision, run every candidate approach against it, and let the numbers decide.
How do MCP and tool-calling change this decision?
Tool-calling (including MCP-based tool use) is a fourth axis that intersects with all three. When your AI feature needs to take actions or query live systems, tool-calling is the retrieval mechanism, not RAG. You do not embed and retrieve database rows; you give the model a tool that queries the database directly. Prompting still governs when and how the model calls tools. Fine-tuning can improve tool selection consistency for complex multi-tool workflows. Think of tool-calling as dynamic RAG with side effects, and apply the same observability discipline: log every tool call, its inputs, and its outputs.
What to Do Next
The decision is not RAG versus fine-tuning versus prompting. It is: what is the actual gap, what is the cheapest technique that closes it, and how will you measure whether it worked. Most teams need better prompts and a retrieval layer. A minority need fine-tuning for a specific sub-task. Almost no one needs to start with fine-tuning.
If you are working through this decision for a real feature or system and want a second opinion from someone who has shipped production AI across multiple stacks, I offer focused AI Architecture advisory engagements. You can also reach out directly if you want to describe the specific problem you are trying to solve before committing to anything. I am solo and independent, which means you get direct advice without an agency markup or a sales funnel disguised as a discovery call.
