The Decision Tree: Prompt, RAG, Tune, Train

Fine-tuning sits in a stack of options. The honest cost-ordered ladder is: prompt engineering, retrieval-augmented generation, fine-tuning, distillation, and from-scratch training. You should walk up that ladder, not skip rungs. Skipping rungs is the most expensive mistake teams make with this technology.

• Prompt first: better instructions, examples that match production distribution, structured output schemas, and a stronger base model fix 60-70% of "we need fine-tuning" requests
• RAG next: if the problem is missing or stale knowledge, fine-tuning will not save you. Vectors and retrieval will
• Fine-tune when: the desired behavior is consistent (a style, a format, a refusal policy, a domain idiom) and prompting cannot encode it cheaply per call
• Fine-tune when: you have a quality bar prompting cannot meet on a smaller model, and the smaller model is needed for latency or cost
• Distill when: you have a frontier model solving a task at acceptable quality but unacceptable cost, and you can label thousands of high-quality input-output pairs
• Train from scratch never, unless you are a frontier lab. The marginal value of a custom pretrain over fine-tuning Llama, Qwen, or Mistral is negative for almost every business
• A useful heuristic: if you cannot describe the bad behavior in three sentences and produce 200 hand-labeled examples of correct behavior, you are not ready to fine-tune

Full Fine-Tuning, LoRA, QLoRA, and the Cost Curve

Parameter-efficient fine-tuning is the default. Full fine-tuning updates every weight in the base model and produces the highest ceiling, but it is rarely the right tradeoff. LoRA freezes the base and inserts small low-rank adapter matrices, training maybe 0.1 to 1 percent of total parameters with roughly 90 percent of the quality of full FT. QLoRA quantizes the frozen base to 4-bit and trains LoRA adapters on top, making 70B-class models trainable on a single H100. The adapter files are tiny, often under 100 MB, which changes deployment economics.

• Full fine-tuning: every weight updated, highest ceiling, highest cost. Justified mostly for safety, refusal, or deep behavior changes
• LoRA: train low-rank adapters, 10x cheaper than full FT, ~90% of the quality on most tasks
• QLoRA: 4-bit base + LoRA adapters, fine-tunes 70B models on a single 80GB GPU. Default for cost-sensitive teams
• Realistic 2026 cost: QLoRA on a 7B model is 2-4 hours on an A100 (~$2-5). 70B QLoRA on an H100 is 8-12 hours (~$15-25)
• Adapter swapping: vLLM, TGI, and SGLang load many LoRA adapters against one base, near-zero per-tenant overhead at inference
• DoRA, IA3, and other PEFT variants exist but LoRA/QLoRA remain the right defaults in 99% of cases
• Full FT comes back into play only when adapters demonstrably underfit on rigorous evals, never as a first move

Supervised Fine-Tuning, Preference Tuning, and Distillation

The vocabulary of training has consolidated around three patterns. Supervised fine-tuning (SFT) trains on labeled input-output pairs. Preference tuning (DPO, KTO, ORPO, IPO) trains on pairs labeled "this response better than that response," teaching the model relative judgment. Distillation transfers behavior from a large teacher into a smaller student through synthetic data generation. Most production pipelines use SFT for capability and DPO for polish, in that order.

• SFT: labeled pairs of {prompt, ideal response}, teaches direct behaviors and formats
• DPO (Direct Preference Optimization): pairs of {prompt, chosen, rejected}, has largely displaced RLHF for alignment in production
• KTO, ORPO, IPO: DPO variants with different tradeoffs around binary labels, reference models, and overfitting
• RLHF still appears in frontier lab pipelines but is overkill for most product teams - infrastructure cost rarely worth it
• Distillation: prompt a frontier model on your distribution, filter outputs by an LLM judge or human, train a smaller open model on the result. Standard playbook to cut inference cost 10-50x
• Synthetic data is fine, but distribution matters more than volume - 5,000 examples that match production beats 100,000 generic examples
• Continuous fine-tuning (training on user-corrected outputs over time) needs strict provenance and PII controls or it will leak and rot

Dataset Engineering: Where the Quality Actually Comes From

The hidden 80% of fine-tuning is dataset construction. Teams obsess over hyperparameters and ignore that their training set is unbalanced, contains label noise, mixes incompatible styles, or has no held-out evaluation slice. A clean, opinionated 1,000-example dataset routinely beats a noisy 100,000-example dataset on the same base model.

• Start by labeling 200 examples by hand, end-to-end, before any tooling. This catches schema and instruction problems early
• Curate ruthlessly: drop ambiguous, duplicated, contradictory, or off-distribution examples. Quality beats volume
• Cover the long tail: explicitly include edge cases, refusal cases, multi-turn cases, and adversarial inputs
• Hold out a real eval set before any training - never train on the eval set, never re-train against eval performance
• Track provenance for every example: source, labeler, version, license. Required for audit, debugging, and PII response
• Synthetic data is acceptable but must be filtered by a stronger model or a human, and tagged so you can ablate its impact
• Tokenizer matching: ensure your data renders identically through the base tokenizer or you will train on bugs
• Schema discipline: chat templates and system-prompt placement must match exactly what production will use

Hosted Fine-Tuning vs Self-Hosted

You can fine-tune through a vendor (OpenAI, Anthropic, Google, Together, Fireworks, Replicate) or self-host on rented GPUs (RunPod, Modal, Lambda, AWS, GCP). Hosted is simpler, faster to first result, and locks you to one provider. Self-hosted gives portability and lower marginal cost at volume but requires real engineering. The right choice depends on volume, sensitivity, and how much you trust the vendor over a 24-month horizon.

• OpenAI fine-tuning: cleanest UX, locked-in inference, useful for prototyping but expensive at scale
• Anthropic and Google: hosted fine-tuning available, similar trade-off. Inference stays inside their stack
• Together, Fireworks, Replicate, Modal: open-weights focused, you keep model artifacts, can move providers
• Self-hosted (RunPod, Lambda, Modal, AWS): full control, lowest cost per token at high volume, real ops cost
• Compliance: regulated workloads (health, finance, defense) usually require self-hosted or single-tenant deployments
• Portability test: can you export the trained weights and run them elsewhere within 48 hours? If no, you are locked in
• Inference stack: vLLM, SGLang, TGI, TensorRT-LLM are the production options. vLLM is the right default in 2026

Evaluation Before and After Fine-Tuning

No fine-tune is real until it passes a frozen eval set. The before-and-after comparison is the only output that matters. Teams that skip rigorous eval ship regressions: a tuned model that beats the base on the obvious cases and silently breaks edge cases that mattered. The eval set is the contract.

• Freeze an eval set before training, drawn from real production traffic with explicit edge cases
• Score the base model first - that is your floor, not zero
• Run identical evals on tuned candidates, compare delta per slice (not just aggregate)
• Include negative tests: things the model should refuse, formats it should reject, sensitive topics
• Trajectory and side-effect evals if the tuned model is part of an agent loop
• LLM-as-judge with a strong judge model (Claude Opus or GPT-5) and rubrics calibrated against human labels
• Tools: DeepEval, Braintrust, Promptfoo, LangSmith, OpenAI Evals, Inspect AI. Pick one and live in it
• Ship gate: tuned model must beat base on every slice that matters, not just average. Regression on any guarded slice blocks release

Production Deployment, Versioning, and Rollback

A fine-tuned model is a versioned artifact like any other dependency. Treat it that way. The teams that move fastest in production are the ones who can roll back a model the same way they roll back code: one command, in seconds, with traffic shifting.

• Version every adapter: training data hash, base model hash, hyperparameter set, eval scores, training run ID
• Canary deployments: route 1-5% of traffic to the new model, watch eval and live metrics for 24-72 hours
• Shadow mode: log new-model outputs alongside production without serving them, compare on real traffic before cutover
• Adapter hot-swap: vLLM with --enable-lora can serve dozens of adapters from one base, useful for per-tenant tuning
• Rollback ready: the previous adapter version must be one config change away, always
• Drift watch: monitor task-level KPIs and eval scores on live samples weekly, fine-tunes can rot as inputs shift
• Cost monitoring: a fine-tune that doubles latency or token usage at inference can wipe out its quality gain economically

When Fine-Tuning Goes Wrong

Most failed fine-tunes are diagnosable in the first week. The failure modes repeat across teams.

• Catastrophic forgetting: tuned model loses general capability because the data was narrow. Mitigate with mixed-domain data or LoRA
• Overfitting to the training distribution: looks brilliant on test, breaks on real traffic. Hold-out eval is the only defense
• Stylistic mimicry without competence: model sounds right but is wrong. Common when training on labels generated by a weaker model
• Hallucinated facts injected into weights: should have been RAG, not training. Now hard to update without retraining
• Cost inversion: paying more to host a custom model than it would have cost to call the larger model. Always do the math first
• No eval baseline: team cannot prove the tune improved anything. The fine-tune becomes a religious belief, not an engineering artifact
• Compounding tuned models on tuned models: each generation shifts the distribution, errors compound, drift accelerates
• Lost portability: tied to one vendor's fine-tuning API with no exportable weights, locked in for the lifetime of the feature

Working With Me on a Fine-Tuning Engagement

Most teams who reach out for fine-tuning help benefit most from a hard look at whether to fine-tune at all. The first deliverable is usually a written recommendation: do this with prompts, do this with RAG, do this with a stronger model, or yes, fine-tune with this dataset, this technique, and this budget. From there, an engagement runs from dataset design through training, evaluation, deployment, and the operational handoff to your team.

• Phase 1 (1-2 weeks): problem audit, current solution review, alternatives analysis, go/no-go recommendation in writing
• Phase 2 (2-4 weeks): dataset design, labeling guide, eval set construction, baseline scoring of current solution
• Phase 3 (2-6 weeks): training runs, hyperparameter exploration, ablation studies, candidate selection
• Phase 4 (1-3 weeks): deployment pipeline, canary, observability, rollback discipline, documentation
• Phase 5 (ongoing): drift monitoring, retraining cadence, evaluation hygiene, knowledge transfer to in-house team
• Deliverables: trained adapters or model weights, training scripts, eval suite, deployment configs, runbook, written postmortem
• Typical engagement: 6-14 weeks for a single high-value fine-tune from problem to production

FAQ