What an LLM Consultant Actually Does

The work lives where the model choice meets the system around it. Most LLM systems do not fail because the model was wrong; they fail because the evaluation harness was missing, the retrieval was naive, the prompt was untested across the customer distribution, or the cost model never got instrumented. The deliverable is a system that ships, holds quality under load, and is cheap enough to expand. Everything else, the model selection, the prompt design, the fine-tuning call, is in service of that.

• Model selection across OpenAI, Anthropic, Google, Mistral, Cohere, and the open-source frontier (Llama, Qwen, DeepSeek, Mixtral)
• Multi-provider routing strategy: which model handles which call, when to fail over, how to A/B against a frontier release
• Prompt design patterns, system prompt versioning, structured-output schemas, and prompt-level regression tests
• Evaluation harness design: rubric authoring, golden datasets, regression sets, drift detection, LLM-as-judge with calibration
• Retrieval architecture: chunking, embedding model choice, hybrid lexical-plus-vector, reranking, query rewriting, freshness handling
• Fine-tuning and adapter strategy: when LoRA, QLoRA, or full fine-tuning beats a better base model with a better prompt
• Cost, latency, and reliability: cost-per-call instrumentation, p50 and p95 latency budgets, streaming, caching, fallback chains
• Production guardrails: PII handling, prompt injection defense, output validation, abuse detection, audit logging
• Vendor contract review: training-data clauses, data residency, rate limits, sub-processor disclosure, IP indemnification

When You Need an LLM Specialist, Not Just an Engineer

The trigger is usually a quality, cost, or reliability problem that the existing engineering team has tried and failed to fix. They are smart and senior, but LLM systems have idiosyncrasies that take years to internalize: how rapidly the model market moves, how to build an evaluation harness that actually catches regressions, why prompt engineering plateaus and when to switch to fine-tuning, how retrieval quality dominates model quality once the prompt is good enough.

• A production LLM feature is shipping but quality is unpredictable and there is no evaluation harness
• Cost per call has tripled over six months because the team kept adding chained calls without instrumentation
• A new model release just dropped and the team has no playbook for evaluating whether to switch
• Retrieval quality is mediocre and the team has tried three vector stores without diagnosing the actual failure mode
• A fine-tuning decision is being debated internally and the engineering team and the data science team disagree
• Prompt injection or jailbreaks have surfaced in production and the team needs a defense posture, not a one-off patch
• The model bill has crossed $50K per month and the CFO is asking how it gets to $10K without losing quality
• A regulator, auditor, or enterprise customer has asked for documentation on the LLM stack the team cannot produce
• Latency has become the bottleneck for adoption and the team is debating streaming, caching, and routing without a clear plan

Model Selection in a Fast-Moving Market

The model market moves faster than any other layer of the stack. Frontier releases hit every quarter from OpenAI, Anthropic, Google, and the open-source frontier (Llama, Qwen, DeepSeek). A model that was the best choice six months ago is rarely the best choice today, and switching is harder than the marketing claims. Good model selection is not picking the highest benchmark; it is picking the model that survives your evaluation harness on your distribution at your cost target.

• Start with the evaluation harness, not the model. Until you have a golden set and a rubric, every model decision is folklore
• Test on your customer distribution, not generic benchmarks. Public benchmarks are leading indicators only
• Multi-provider by default: keep at least two providers warm so a regression, outage, or pricing change does not stall the product
• Tier the calls: frontier model for the hard 10%, mid-tier for the routine 70%, small or distilled model for the cheap 20%
• Track total cost of ownership: API price, plus retries, plus tool calls, plus evaluation, plus observability, plus engineer time to migrate
• Open-source is now competitive for many enterprise classes (Llama 3 and 4, Qwen 2.5 and 3, DeepSeek), but inference economics depend on volume and the willingness to operate GPUs
• Version-pin in production. Frontier providers ship silent quality changes; pin to a dated snapshot when reliability matters
• Set a quarterly model review cadence with a documented rubric so the team is not relitigating model choice every Slack thread

Evaluation Harness Design, the Highest-Leverage Work

Production LLM systems live or die by evaluation. Without a real evaluation harness, every change is a vibes-based decision and quality regressions are discovered by customers. The harness is the single highest-leverage investment in a production LLM stack, and it is almost always missing or weak in teams that have not had a senior LLM practitioner involved.

• Golden dataset: 50-500 hand-curated examples covering the customer distribution, edge cases, adversarial inputs, and the long tail
• Rubric: explicit scoring criteria per output dimension, calibrated against human labels on a holdout set
• LLM-as-judge with calibration: a stronger model scores outputs against the rubric, calibrated quarterly against human review
• Regression sets: every shipped bug becomes a regression test that runs on every change
• Drift detection: production sample of real traffic scored continuously, alerting on quality drop or distribution shift
• Cost-per-quality tracking: regression in quality often shows up first as cheaper calls, before customer complaints surface
• Trajectory-level evaluation for agentic systems: score the path, not just the final output, including tool-choice correctness
• Tooling: LangSmith, Langfuse, Braintrust, Arize Phoenix, Helicone, or homegrown - the choice matters less than having one in place

Retrieval, RAG, and the Architecture That Actually Works

RAG remains the default for any LLM system that needs to ground answers in a specific corpus. The naive implementation, chunk the docs, embed them, top-k cosine similarity, paste into prompt, is a tutorial pattern, not a production pattern. Production retrieval is a system: chunking strategy tuned to the content, hybrid lexical-plus-vector search, a reranker, query rewriting, freshness rules, and an evaluation harness specifically for retrieval quality.

• Chunking matters more than the embedding model. Recursive, semantic, parent-document, and contextual retrieval all beat fixed-size chunking on most corpora
• Hybrid retrieval is the modern default: BM25 plus vector, fused with reciprocal rank fusion
• Rerankers (Cohere, Voyage, BGE, or fine-tuned cross-encoders) are the single most cost-effective quality lever after chunking
• Query rewriting and HyDE handle the gap between user phrasing and corpus phrasing
• Freshness and recency rules: most production corpora have a recency dimension the naive embedding model ignores
• Retrieval evaluation is separate from end-to-end evaluation: measure recall at k on a labeled retrieval set, not just downstream quality
• Vector store choice (Pinecone, Weaviate, Qdrant, pgvector, Turbopuffer, Milvus, Vespa) matters less than chunking and reranking
• Multi-hop and agentic retrieval (search, read, search again) often beats single-pass retrieval on complex queries, at proportionally higher cost
• Graph and structured retrieval increasingly complement vector retrieval for entity-heavy corpora

Fine-Tuning, LoRA, and When Prompts Are Enough

Fine-tuning is romanticized in the market and oversold by vendors. In 2026, the practical sequence is prompt, then retrieval, then fine-tune, then distill. Most teams jump to fine-tuning prematurely and burn three months on a project a better prompt and a reranker would have solved in a week. Fine-tuning earns its keep when prompt engineering has plateaued, when latency demands a smaller model, when domain vocabulary is too dense for in-context learning, or when the cost arithmetic flips at scale.

• Right sequence: prompt, then retrieval, then fine-tune, then distill. Skipping steps wastes months
• LoRA and QLoRA on a strong base model is the default fine-tuning approach in 2026, not full fine-tuning
• Pair fine-tuning with retrieval rather than replacing it; the highest-ROI adapter teaches style and format, not facts
• Fine-tuning makes sense for tone, format, narrow classification, latency reduction via smaller models, and tasks where the customer distribution is genuinely outside the base model training set
• Fine-tuning does not make sense for keeping the model up to date on facts, customer-specific knowledge that changes weekly, or anything a reranker plus better prompt would solve
• Distillation: train a smaller model on the larger model's outputs to capture quality at cheaper inference, especially valuable above 10M calls per month
• Evaluation matters more in fine-tuning than anywhere else: a fine-tune without an evaluation harness is a regression generator
• Vendor fine-tuning (OpenAI, Anthropic, Google) versus self-hosted fine-tuning (Hugging Face, Unsloth, Axolotl) is a contract and operational decision, not a quality one

Cost, Latency, and Reliability Posture

Most production LLM systems leak money. The cost per call is rarely modeled before launch, retries and chained calls compound, and the bill grows faster than usage. Latency tail (p95, p99) determines adoption more than median latency, and most teams optimize the wrong percentile. Reliability is treated as a model-provider problem until it stops being one in front of a customer.

• Cost-per-call instrumentation tied to feature and customer segment, reviewed monthly with a published trend
• Cost shape: prompt caching, response caching, tool-result caching, semantic caching for repeated queries
• Tiered routing: frontier model for the hard cases, mid-tier for routine, distilled or open-source for cheap volume
• Latency budget: p95 target named per feature, streaming where the UX supports it, parallelism where the workflow allows
• Reliability: multi-provider fallback, retry policy with exponential backoff, circuit breakers, degraded-mode responses
• Quotas and budgets: hard caps per customer, per feature, per call type, enforced outside the model
• Observability: full prompt and response logging with PII redaction, sampled trajectory replay, regression alerting
• Capacity planning: token-per-second forecasts, provider rate-limit headroom, fallback capacity in a second region

Engagement Shapes and Pricing in 2026

LLM consulting engagements come in three common shapes: a focused diagnostic, a project-scoped delivery, or a retainer. Pure day-rate engagements are rare in this space because most useful LLM work needs continuity across a feature lifecycle. The rates below reflect what a senior independent practitioner with significant production LLM experience charges in 2026.

• US hourly: $250-$500/hr for senior LLM specialists, $300-$400/hr is the realistic median
• US day rate: $2,500-$5,000/day, with $3,000-$4,500 standard for hands-on LLM work
• US monthly retainer (2-3 days/week): $30,000-$60,000
• UK day rate: GBP 1,200-2,000/day in London, GBP 900-1,500 outside it
• EU day rate: EUR 1,500-2,800/day in major hubs
• 2-4 week LLM diagnostic engagement: $20K-$60K fixed fee, producing an architecture review, evaluation gap analysis, and cost-reduction plan
• 6-12 week LLM delivery engagement: $60K-$200K, producing a shipped feature plus evaluation harness
• Production RAG application benchmark from industry surveys: $75K-$250K over 8-16 weeks for the full first version
• Red flag: under $200/hr is a mid-career engineer with LLM hobby experience; over $1,200/hr without specific industry depth is selling brand

Red Flags When Hiring an LLM Consultant

The market is full of consultants whose LLM experience is six months of demos. Use the checklist below to filter quickly. The signal is whether the consultant has been on-call for a production LLM system, not how many tutorials they have read.

• Cannot whiteboard a retrieval pipeline including chunking, hybrid search, reranking, and evaluation
• Has no opinion on the prompt-to-fine-tune sequence and treats fine-tuning as a default
• Cannot describe a recent production LLM incident they were involved in resolving
• Brand-loyal to a single model provider without engaging the multi-provider routing argument
• Quotes evaluation as "we use eval" without naming the dataset construction, rubric, or calibration method
• Has no opinion on prompt injection, data exfiltration, or guardrail patterns
• Resells a specific vector store, observability tool, or evaluation platform with undisclosed commercial relationship
• Cannot read the API pricing pages of OpenAI, Anthropic, and Google from memory at the order-of-magnitude level
• Has never personally calibrated an LLM-as-judge against human labels
• Treats latency as a model-provider problem and has no streaming, caching, or routing playbook

FAQ