LLM Application Development - Production-Grade AI Apps

LLM application development is the engineering discipline of shipping production features whose core logic is a call to a large language model. It is broader than agent work. Most AI features in production are not full agents. They are LLM applications that classify, summarize, retrieve, generate, route, extract, score, or rewrite. These features ship inside SaaS products, internal tools, customer support stacks, sales workflows, and data pipelines, and they need the same engineering discipline as any other production code: evaluation, observability, rollback paths, clean integration, and a cost shape your CFO can live with.

This page is for engineering and product leaders deciding who should build their LLM feature. I work as either the senior IC who ships the feature end-to-end, or the architect-plus-reviewer to an internal team. The work usually starts with one question the team has not yet answered: how will you know this feature is working in production. Until you have an evaluation harness, you do not have a product, you have a demo.

What Counts as an LLM Application

An LLM application is any production system whose behavior depends on a call to a language model. The taxonomy below covers what I see in 90% of buyer requests. The category matters because each has a different evaluation profile, latency budget, and cost shape.

Classification and routing: input goes to one of N categories, latency budget tight, eval is accuracy on a labeled set
Extraction: pull structured fields from unstructured input, eval is per-field precision and recall
Summarization: long input to short output, eval is faithfulness plus readability, hard to automate cleanly
Generation: blank-page output (drafts, code, copy), eval is rubric-based LLM-as-judge plus human review
Retrieval-augmented generation (RAG): retrieve then generate, eval is retrieval recall plus generation faithfulness
Rewriting and editing: known input, transformed output, eval is constraint compliance and style match
Scoring and ranking: numeric output for sorting, eval is correlation with human or business outcome
Conversation and chat: multi-turn, eval is harder and usually trajectory-based
Agentic features: model controls tool use and flow, separate category, see Agent Builder

Picking the Right Model

Model choice in 2026 is no longer one-decision. A mature LLM application routes per request based on task complexity, cost, latency, and capability profile. The major frontier model families (Claude Opus/Sonnet/Haiku, GPT family, Gemini, Grok) each have strengths. Open source (Llama, Qwen, Mistral, DeepSeek) is now production-credible for many narrower tasks and gets cheaper every quarter.

Use a frontier model (Claude Opus, GPT, Gemini Pro) for the hard reasoning hops, route easy hops to a small model
Cache aggressively: prompt caching on Anthropic and OpenAI cuts repeat-context cost by 80-90%
Structured outputs (JSON Schema, function calling) are mandatory anywhere downstream code parses results
Open source via Together, Fireworks, Groq, or self-hosted (vLLM, TensorRT-LLM) is competitive for classification and extraction
Fine-tuning is rarely the right answer first: prompt and RAG first, fine-tune only when evals plateau and the data exists
Capability gaps matter: vision, long context, tool use, structured output, safety profile all vary by family
Cost shape: input vs output token pricing, cache discounts, batch API discounts, all factor into per-request economics
Always have a fallback model: every production app needs a secondary provider for outages

Prompt Engineering Worth The Name

Prompt engineering in 2026 is software engineering. Prompts are versioned in source control. Changes go through code review. Every prompt change runs a regression evaluation suite before deploy. Teams that treat prompts as wiki pages or magic strings break production weekly. Teams that treat prompts as code ship reliably.

Prompts live in source control, versioned with the code that calls them
One prompt file per template, with metadata: model, temperature, max tokens, expected output schema
Variables are interpolated through a typed template engine, not string concatenation
Every prompt change triggers an eval run on the regression suite before merge
Prompt management platforms worth naming: Langfuse, LangSmith, PromptLayer, Helicone, Confident AI
Branching and A/B testing built into the prompt platform, not the application code
System prompts loaded from disk, never inlined in business logic
Few-shot exemplars maintained as a separate dataset, refreshed when the data distribution drifts
Output parsing is its own layer with strict schemas and graceful failure

Retrieval-Augmented Generation, Done Properly

Most production LLM applications include retrieval. The naive RAG architecture (chunk, embed, top-k, stuff in prompt) works for prototypes and fails for serious products. Senior RAG work is the chain of decisions about chunking strategy, embedding choice, hybrid search, reranking, query rewriting, and the evaluation harness that catches when each link in that chain regresses.

Chunking strategy matters more than embedding choice: sliding window, sentence-aware, layout-aware, table-aware
Hybrid search (vector + BM25) outperforms pure vector on almost every benchmark, ship it by default
Rerankers (Cohere Rerank, BGE Rerank, Voyage Rerank) cut prompt size by 5-10x with better relevance
Query rewriting and decomposition: LLM rewrites the user query into something the retriever can actually answer
Metadata filtering on the index: tenant, date, document type, never trust the LLM to filter via prompt instruction
Citation enforcement: the answer must reference the chunks it used, both for users and for evaluation
Retrieval evals are separate from generation evals: measure recall@k on a labeled dataset
Generation evals: faithfulness, groundedness, context relevance via TruLens, Ragas, or your own LLM-as-judge
Knowledge graph hybrids (GraphRAG, Microsoft research) for relational reasoning that vector search cannot handle

Evaluation: The Difference Between A Demo And A Product

Without an evaluation harness, you cannot tell whether a prompt change made things better or worse, whether the new model is actually an upgrade for your task, or whether last week production deploy regressed quality. Evaluation is the missing leg under every flaky LLM product. I build the eval harness before I build the feature.

Build the eval set first: 20-100 hand-labeled examples that span the input distribution
Define metrics tied to the business outcome, not vanity metrics
LLM-as-judge with explicit rubrics, calibrated against human labels on a holdout set
Regression suite runs on every prompt or model change, blocking deploy if quality drops
Online evaluation: sample production traffic, score continuously, alert on drift
Cost and latency evaluated alongside quality, treat all three as constraints
Tools worth naming: LangSmith, Langfuse, Braintrust, Confident AI DeepEval, Ragas, TruLens, OpenAI Evals, MLflow
Pin a golden dataset and re-evaluate every quarter as models and prompts drift

Observability, Cost, And Latency

In 2026 LLM observability is a $2.69 billion market and Gartner is forecasting half of GenAI deployments will use it by 2028. Translation: this is no longer optional. Every LLM call in production must be traced, every token must be counted, every error must be classified, and every regression must be alertable. Without this layer, your AI feature is a black box that costs unbounded dollars.

Full request tracing: input, output, model, latency, tokens in, tokens out, cost, parent trace ID
Tracing platforms: LangSmith, Langfuse, Helicone, Portkey, Braintrust, Arize Phoenix, MLflow LLM
AI gateways (Helicone, Portkey, OpenRouter) for routing, caching, fallback, and centralized cost tracking
Cost budgets per feature, per tenant, per user, enforced upstream of the model call
Latency budgets enforced via timeouts and streaming UX
Streaming responses: first token under 500ms is the user-perceived bar
Failure classification: provider error, schema violation, content policy, quota, downstream
Drift detection on inputs and outputs: alert when distribution shifts

Production Patterns That Actually Hold Up

A handful of patterns separate LLM features that ship and stay shipped from features that break weekly.

Structured outputs with strict schemas, validated before downstream code touches them
Graceful degradation: schema failure falls back to a smaller model or a rule-based path
Prompt versioning in git, with semantic version numbers tied to eval scores
Per-tenant prompt overrides for enterprise customers without forking code
Caching everything cacheable: prompt cache, response cache, embedding cache
Idempotency on any write-side LLM action: retries must not duplicate
Rate limiting per tenant and per user, well below provider limits, to keep one bad actor from starving others
PII redaction before the model call, when you cannot trust the data path
Audit log of every model call for regulated environments
Feature flags for prompt and model rollout, never deploy a prompt change to 100% of traffic at once

What I Ship In An Engagement

A typical LLM application engagement is two to ten weeks. It can be a single feature inside an existing product, a new product surface, or a senior review and rebuild of an existing feature that is misbehaving. I work in TypeScript and Python by default, and I integrate cleanly with Node, Next.js, Python FastAPI, Django, Rails, Go, and most managed Postgres or vector stores.

Week 1: task definition, eval set construction, baseline measurement, architecture brief
Week 2: prompt or RAG implementation, observability instrumentation, regression eval running
Week 3-4: iterate against the eval harness, hit target metrics, tune cost and latency
Week 5-6: hardening, on-call runbook, feature flags, pilot rollout
Deliverables: the feature, the eval harness, the observability dashboards, the prompt registry, the runbook
Code written in your stack, with your patterns, your tests, your CI, designed for your team to extend
Optional advisory tail: 2-4 hours per month for the next quarter to support the team

FAQ

Is an LLM application the same as an AI agent?

No. An agent is a system where the LLM controls the flow of execution: it picks which tool to call and when to stop. An LLM application is anything broader, including classification, extraction, summarization, RAG, generation, and routing. Most production AI features are LLM applications, not full agents.

Which model should I use?

Route per request. Use a frontier model (Claude Opus, GPT, Gemini Pro) for hard reasoning, a small model (Haiku, GPT smaller tiers, open source) for easy hops. Cache aggressively. Always have a fallback provider for outages.

Do I need RAG?

You need RAG if the model has to ground its answer in your data that is not in its training set. You do not need RAG if the task fits in the model context window directly, or if your data is fully fine-tuned in. Most production LLM apps with internal data do need RAG.

How do I evaluate an LLM feature?

Build a 20-100 example labeled set tied to the business outcome. Define metrics. Run an LLM-as-judge with rubrics calibrated against human labels. Make the eval suite a CI gate on prompt and model changes. Sample production traffic for online evaluation. Use LangSmith, Langfuse, Braintrust, Confident AI, Ragas, or TruLens.

How much does an LLM feature cost to run?

Highly variable. Per request, you can land anywhere from sub-cent (cached, small model, structured output) to multiple dollars (large model, long context, no cache). The honest answer requires per-feature analysis. Cost is a first-class constraint, not an afterthought.

How long does an LLM application take to ship?

Two to ten weeks for a feature of moderate scope. One to two weeks of discovery and eval setup, three to six weeks of build and iterate, one to two weeks of hardening and rollout. Larger features and regulated domains go longer.

Should I fine-tune the model?

Rarely as the first move. Prompt engineering and RAG solve 80% of use cases. Fine-tuning is right when evals plateau, you have the data, and the task is narrow enough that the cost is justified. For most teams, fine-tuning is a phase 2 or 3 decision, not a phase 1.

Do I need a separate observability platform or is logging enough?

You need a separate platform. Logs do not capture token costs, latency per node, prompt versions, eval scores, or drift, which are the things that break LLM products in production. LangSmith, Langfuse, Helicone, Braintrust, and Arize Phoenix are the major options.

Next step

Your situation isn't generic.
Neither should the conversation be.

A short call to map what llm application development looks like for your team. No obligation, no pitch, just clarity.

Build my LLM application See Agent Development

Senior architect · 16+ years shipping · Direct, no agency layers

LLM Application Development