Capability Tiers and What They Mean in Practice

The market has converged on three or four useful tiers. Confusing them is the most common selection mistake. A frontier model on a simple classification task is wasted money. A fast tier model on a long reasoning chain is a quality regression.

• Frontier reasoning: GPT-5.5 thinking, Claude Opus 4.7 with extended thinking, Gemini 3.1 Pro deep think, Grok 4. Long context, hard reasoning, multi-step planning
• Frontier general: GPT-5.4, Claude Sonnet 4.6, Gemini 3.1 Pro. Near-frontier capability, much cheaper than thinking tiers, good default for general agent and chat work
• Fast / cheap: GPT-5 mini, Claude Haiku 4.5, Gemini 3.5 Flash, Gemini Flash-Lite. Classification, extraction, routing, simple chat
• Open weights frontier: DeepSeek V4, Qwen 3, Llama 4 405B. Near-frontier quality at 5-70x lower API cost, self-hostable
• Specialist: code models (e.g. tuned Sonnet, Qwen Coder), embedding models, image-output models. Picked per modality, not per benchmark average
• Reasoning-on-demand: GPT-5.4 and Claude Sonnet 4.6 with optional thinking mode let you toggle reasoning per request. Powerful for cost-sensitive routing

The Pricing Reality in 2026

Headline benchmark scores hide an order of magnitude in cost. The same task can cost $0.50 or $50 per thousand calls depending on which tier you pick. The cost equation has three terms: input price, output price, and effective tokens used per request (which thinking models inflate dramatically).

• GPT-5.4: ~$2.50 input / $15 output per million tokens. Pro tier ~$30 / $180
• Claude Sonnet 4.6: near-frontier with 1M context and strong prompt caching, mid-tier pricing
• Gemini 3.5 Flash: ~$0.15 input / $0.60 output. Cheapest credible flagship-class option from a major US provider
• Gemini Flash-Lite: ~$0.10 / $0.40, cheapest proprietary frontier-adjacent tier
• DeepSeek V4: ~$0.43 input / $0.87 output. 5-70x cheaper than US frontier on comparable benchmarks
• Prompt caching: Anthropic and OpenAI both offer it. Cuts repeated input cost 50-90% on stable system prompts
• Thinking-mode billing: extended thinking tokens are billed as output. A 2000-token thinking trace at GPT-5.5 Pro is ~$0.36 per call. Cap and audit
• Egress and gateway markup: Bedrock, Azure OpenAI, Vertex AI add 0-15% over native pricing for compliance and SSO

Benchmarks, and Why You Should Distrust Them

MMLU, GPQA, SWE-bench, Arena Elo, Humanity's Last Exam, and the rest of the leaderboard set are useful for filtering candidates but not for picking one. They measure general capability, not your task. Two models that score within a point on MMLU can differ 20% on your domain. Always run your own task evals on a shortlist of 3-5 models before committing.

• Public benchmarks: filter the shortlist, never the final selection
• Arena Elo (LMArena, formerly Chatbot Arena): reasonable proxy for "feel good" general use, weak for narrow tasks
• SWE-bench Verified: best public proxy for code editing capability. Take with a grain of salt - serious training contamination risk
• GPQA Diamond and Humanity's Last Exam: hardest reasoning. Useful to separate the top tier from everything else
• Your own eval set is the only one that matters - 50-200 real examples from your distribution, scored on the actual deliverable
• Benchmark gaming is real: providers train against public sets. Score your candidates on private examples they have never seen
• Run latency and cost benchmarks alongside quality - a 5% quality win that costs 10x is usually not worth shipping

Routing: The Pattern That Pays for Itself

A single model serving all traffic is almost always wrong. The 80/20 of LLM cost engineering is routing: send each request to the cheapest model that can handle it, and escalate when it cannot. Routing is technically simple and economically dramatic. Cost reductions of 50-90% on previously frontier-only stacks are routine.

• Classifier-routed: a small model classifies the request and picks the target tier. Adds a few hundred milliseconds, saves orders of magnitude
• Confidence-routed: cheap model runs first; if its confidence (logprobs, self-rated, or eval-derived) is low, escalate to a stronger model
• Capability-routed: by task type (code, chat, extraction, reasoning), each routed to a model picked specifically for it
• Budget-aware routing: enforce per-user, per-tenant, per-feature cost caps. Downgrade gracefully when caps hit
• Fallback chains: primary, secondary, tertiary models with automatic failover on rate limit, outage, or refusal
• Tools: OpenRouter, Portkey, LiteLLM, Vercel AI Gateway, and provider-native routers (Bedrock, Vertex). Pick one and standardize
• Anti-pattern: hard-coded model name in application code. Always go through a routing layer

Open Weights vs Hosted

The open-weights ecosystem has caught up. DeepSeek V4, Qwen 3, Llama 4, and Mistral now sit within a few points of frontier on many benchmarks at a fraction of the cost. The decision is no longer about quality alone. It is about portability, compliance, and how much engineering capacity you have to operate inference yourself.

• Hosted (OpenAI, Anthropic, Google, xAI): lowest engineering cost, latest models first, vendor lock on weights and price
• Hosted open-weights (Together, Fireworks, Groq, Cerebras, Replicate): get open models as an API, keep portability across providers
• Self-hosted on cloud GPUs (AWS, GCP, Azure, RunPod, Modal, Lambda): full control, lowest cost per token at high volume, real ops cost
• On-prem: only when compliance demands it (defense, regulated health, government). Multi-quarter engineering investment
• Compliance: SOC2, HIPAA BAA, data residency, EU AI Act - vendor list narrows quickly when these are non-negotiable
• Quality gap: closing fast on average tasks, still real on hardest reasoning and tool use - test your task before committing
• Portability test: can you swap providers in 48 hours? If no, your selection has hidden lock-in

Latency, Throughput, and Streaming

Quality and cost dominate selection conversations. Latency usually does not - until production traffic hits and the feature feels slow. The four numbers that matter: time-to-first-token, tokens-per-second, total wall time, and tail latency at p95 / p99.

• Time-to-first-token (TTFT): what users perceive as responsiveness. Stream output to mask total latency
• Tokens-per-second (TPS): determines wall time on long outputs. Groq, Cerebras, and SambaNova lead by order of magnitude on open weights
• p95 and p99 latency: more important than average. Providers can have wide tail distributions especially at peak
• Reasoning models add 5-60 seconds of thinking before output - design UX (intermediate states, streaming traces) accordingly
• Batch APIs (Anthropic batches, OpenAI batch, Vertex batch) cut cost ~50% if your task tolerates async
• Edge proximity: Cloudflare Workers AI, Vercel AI SDK on edge cut TTFT by 100-300ms for global users
• Always measure on your own traffic - vendor latency numbers are best-case in their datacenter

Selection Criteria That Actually Matter at Scale

Beyond the tier and price, the criteria that decide whether a model works at scale are operational. Most failed model selections fail on one of these, not on capability.

• Rate limits and capacity: ask for your projected peak QPS in writing. Default quotas trip the first time you launch
• Tool use quality: function calling reliability varies by model. Test with your actual tool surface
• Structured output enforcement: not all providers offer strict JSON schema mode. Critical for production
• Context window: nominal vs effective. Most models degrade past 50-75% of nominal context
• Prompt caching support: huge cost lever on stable prefixes. Anthropic and OpenAI lead, others catching up
• Refusal rate on your domain: legal, medical, security tasks see big differences in over-refusal between models
• SLA and incident history: check status pages and incident frequency, not vendor marketing
• Roadmap risk: vendors deprecate models on 6-12 month windows. Plan for replacement before launch, not after

What an Engagement Looks Like

A model selection engagement typically takes 2-6 weeks and produces a written decision document, a routing implementation, and an evaluation harness the team uses to re-evaluate quarterly. The deliverable is not "you should use X." It is "here is how to choose, here is the current answer, here is how to change the answer next quarter without rewriting everything."

• Week 1: workload audit, current cost and quality baseline, shortlist of 4-8 candidate models
• Week 2-3: eval set construction from real traffic, scoring each candidate on quality, cost, latency, refusal rate
• Week 4: routing layer design and implementation, fallback policy, budget caps, observability
• Week 5-6: production rollout, A/B testing, monitoring, documentation handoff
• Deliverables: written selection memo, eval harness, routing layer code, cost/quality dashboard, quarterly review playbook
• Typical outcome: 40-80% cost reduction with quality flat or improved, plus the ability to swap models in days not months

FAQ