Which Model Should You Use for Your AI Agent? Latency, Cost, and Capability

Which LLM Should You Use for Your AI Agent?

Use the cheapest model that is capable enough for each specific role in your agent: route and classify with a fast, cheap model like Haiku, call tools and structure outputs with a mid-tier model like Sonnet, and reserve the expensive frontier model for multi-step reasoning tasks that actually need it. One frontier model running every step of your agent is almost always the wrong architecture.

I am Mahmoud Zalt, an independent senior AI systems architect with 16-plus years building production software. I founded Sista AI, where choosing the right model for each autonomous agent has been a constant, cost-sensitive decision across a year of production operation. I design and build production AI agent systems for companies that need real results, not demos. If your agent is costing too much or performing inconsistently, see my AI Agent Development services or read more about my work.

Why 'Just Use the Best Model' Is an Expensive Mistake

The reflex is understandable: you want the best output, so you reach for the most capable model. The problem is that a frontier model like Claude Opus 4.8 costs $5 per million input tokens and $25 per million output tokens. A task like 'is this user message a question about billing or a question about features' does not require that level of capability. Claude Haiku 4.5 costs $1 per million input tokens and $5 per million output tokens, and it handles classification, routing, and simple extraction with high accuracy.

In a production agent handling 100,000 requests per day, the difference between routing every message through Opus versus Haiku is several thousand dollars per month. Worse, the latency on your fast-path steps doubles or triples for no quality gain. The correct mental model is not 'which model do I use' but rather 'which model do I use for each role in the pipeline.'

These prices are from the Anthropic API as of mid-2026. The ratio matters more than the absolute numbers: Opus costs five times more per input token than Haiku. Using Opus to classify intent is like hiring a principal engineer to sort your inbox.

The Three Functional Roles Every Agent Has

Every non-trivial AI agent has at least three functional roles, whether you have modeled them explicitly or not. Getting model selection right means treating each role separately.

Role 1: The Router

The router reads incoming user input and decides what category it falls into, what intent it represents, or what next step the system should take. This is a classification problem. The answer space is small and bounded. You have written the categories yourself. A fast, cheap model with a tight prompt handles this reliably. Routing is where you spend Haiku tokens, not Opus tokens. A good router prompt is specific: give it the list of categories, a one-sentence description of each, and a worked example. Temperature zero, short output. Latency under 200ms is achievable.

Role 2: The Tool-Caller

The tool-caller receives structured context, decides which tools to invoke, formats the invocations correctly, and integrates the results. This requires more capability than routing but less than deep reasoning. The model needs to read an API schema, pick the right function, and fill in the parameters correctly. Sonnet-class models do this well. The key constraint is that the model must follow the schema reliably, which means structured outputs with strict validation, not hoping the model writes valid JSON under pressure. At this role, errors in tool calling are expensive: a bad parameter sent to a payment API or a database write call has real consequences.

Role 3: The Reasoner

The reasoner faces open-ended problems: write a plan, diagnose a complex issue across multiple data sources, synthesize conflicting information, produce a long-form deliverable. This is where frontier-model capability pays for itself. On Opus 4.8 with adaptive thinking enabled, the model can reason across a 1M-token context, maintain coherence across dozens of tool calls, and self-correct. The cost is justified because the output quality differential is measurable. Do not use Opus for routing. Do use Opus for tasks where a lesser model would fail or require five retries.

Worked Example: A Customer Support Agent

Here is how I would structure model selection for a customer support agent that handles billing, technical issues, and feature requests for a SaaS product.

Step 1: Route the incoming message (Haiku 4.5)

The first call is cheap and fast. Input: the user message plus a short system prompt listing the categories. Output: one of billing, technical, feature_request, or escalate. Token cost per call: roughly 200 input tokens, 5 output tokens. At Haiku pricing, this is fractions of a cent per message.

Step 2: Decide tool calls (Sonnet 4.6)

Once you know the category, Sonnet receives the message, the relevant tools for that category (account lookup, ticket history, knowledge base search), and the extracted intent. It decides which tools to call and in what order. You use structured outputs here: the response schema is a list of tool invocations with typed parameters. Strict validation catches hallucinated parameters before they hit your backend. If a tool result requires a follow-up tool call, Sonnet handles that loop. You budget roughly 1,000 to 2,000 input tokens per turn here.

Step 3: Generate the final response (Sonnet 4.6 or Opus 4.8)

For straightforward cases (billing inquiry with clean account data, a known technical issue with a documented fix), Sonnet drafts the response. For escalated cases, complex multi-system diagnosis, or anything requiring judgment across conflicting signals, Opus takes the tool results and produces the final answer. This model selection is dynamic: a simple flag on the route output tells you whether to promote to Opus.

The resulting architecture costs roughly 80 percent less per request than running Opus for every step, while the reasoning quality on hard cases is identical to a single-model Opus setup, because hard cases still go to Opus.

Latency, Cost, and Capability: The Real Tradeoffs

The three variables interact in ways that trip up most teams building agents for the first time.

Latency is cumulative

If your agent makes five sequential model calls, user-perceived latency is the sum of all five. A pipeline where every step goes to Opus at 3-5 seconds per call will feel broken to users expecting sub-second responses. Fan out parallel calls where possible, and reserve sequential calls for steps that genuinely depend on the previous output. Haiku is fast enough for synchronous routing. Sonnet is fast enough for single-turn tool selection. Opus is best reserved for async tasks where the user expects to wait, or where you stream the output.

Cost compounds with volume

At low volume, model cost is negligible. At 10 million requests per month, the difference between Haiku and Opus on a routing step is tens of thousands of dollars. Build cost instrumentation from day one: log input and output tokens per step, per model. When usage grows, you will know exactly where to optimize. The common mistake is to optimize later, then discover that a single poorly-scoped Opus call is consuming 60 percent of the API budget.

Capability has diminishing returns

For most structured tasks, Sonnet reaches 95 percent of Opus quality at 60 percent of the cost. The remaining 5 percent matters for complex reasoning, long-horizon planning, and nuanced judgment. Know which of your agent steps require that 5 percent. Most do not. Run evals on a representative sample of your task distribution before you commit to a model tier. Do not assume frontier capability is necessary because the task sounds hard. Test it.

Evals, Guardrails, and Observability in a Multi-Model Agent

Multi-model architectures add complexity to quality assurance. Here is what I apply in production systems.

Evals per role, not just end-to-end

End-to-end evals are necessary but not sufficient. A routing failure early in the pipeline produces a downstream failure that looks like a reasoning failure. Build role-specific evals: a routing eval with labeled test cases, a tool-calling eval with expected API payloads, a final-response eval with human-rated quality scores. When something breaks, you know which layer to fix.

Guardrails at the handoff points

The most dangerous moment in a multi-model agent is the handoff between the tool-caller and the downstream system. Validate tool call parameters against a schema before sending them. For destructive operations (writes, deletes, payment actions), require an explicit confirmation step or a human-in-the-loop gate. On Sonnet 4.6, use strict: true on your tool definitions to get schema-validated inputs with additionalProperties: false enforced. This is not optional for production systems.

Observability by model call

Log the model used, input tokens, output tokens, latency, and a trace ID on every call. Group by role in your dashboards. When the routing model starts producing unexpected categories (distribution shift in user input), you will see it as an anomaly in your routing-tier metrics before it surfaces as user complaints. Prompt caching with cache_control markers on stable context blocks reduces both cost and latency on repeated calls and gives you cache hit rate as an additional health signal.

Human-in-the-loop placement

Place human review at the highest-stakes decision point, not at every step. In most agent architectures, that means reviewing the final action before it executes, not reviewing every reasoning step. Tool-calling agents with always_ask permission policies let you interpose a confirmation event on specific tools without blocking the rest of the pipeline.

Retrieval and MCP: Context Shapes Model Selection

The model you need is partly a function of what context is available to it. A well-designed retrieval layer changes which tier is sufficient for a given task.

Retrieval reduces required model capability

If your agent needs to answer questions about a 500-page technical manual, the naive approach is to send the whole document to a frontier model with a large context window. A better approach is a retrieval-augmented setup: retrieve the three most relevant chunks, pass those to a mid-tier model. The smaller context, cleaner signal, and simpler reasoning task means Sonnet handles it reliably where Opus seemed necessary. Good retrieval does not just reduce cost; it improves accuracy by eliminating distractor content.

MCP for tool-calling at scale

The Model Context Protocol standardizes how agents discover and call external tools. In an MCP-connected agent, the tool-calling role can query the available tool set dynamically rather than loading every schema up front. This matters for model selection because large tool sets bloat the context and push you toward frontier models that handle more tokens gracefully. Tool search with deferred loading keeps the active context small and lets Sonnet-class models perform well even when the total available tool surface is large. Declare tools on the agent definition, use mcp_toolset references, and load schemas on demand.

Context budget and model tier interact

A routing call should have a tight context budget: the system prompt, the user message, nothing else. A reasoning call can have a large context: all tool results, conversation history, retrieved documents. Match the context budget to the role. A model forced to reason over a bloated context with irrelevant information performs worse than the same model given clean, scoped input. Prompt caching on stable context (the system prompt, tool schemas, reference documents) makes large-context calls cheaper without requiring architectural changes.

What Teams Get Wrong When Choosing a Model

After building AI agent systems for companies across industries, I see the same mistakes repeatedly.

Using frontier models because the demo felt better. Demos are not evals. A frontier model produces more impressive output in a demo because it elaborates more. In production, elaboration is often noise. Build a proper eval on real user inputs before committing to a model tier.

Mixing model tiers without clear contracts. If your router runs Haiku and your reasoner runs Opus, the output of the router becomes the input to the reasoner. A vague or ambiguous routing output causes downstream failures that are hard to attribute. Define strict output schemas at every role boundary. The router should return a typed enum, not a free-text description of its decision.

Ignoring the effect of adaptive thinking on cost. On Opus 4.8 with thinking: {type: 'adaptive'}, the model decides how much to think per request. For tasks where deep reasoning is not needed, the model uses little thinking budget. For hard tasks, it uses more. This is the right default for a reasoning role. However, if you use adaptive thinking on a tool-calling role that does not need it, you pay for thinking tokens that do not improve output quality. Scope adaptive thinking to roles where reasoning depth varies by input.

Not accounting for prompt caching in cost models. With cache hits on a stable system prompt, effective input token cost drops by roughly 90 percent on cached portions. A Sonnet call with a large cached context can cost less than a Haiku call without caching. Model your actual cost including cache hit rates, not just nominal per-token prices.

Treating security as an afterthought. A multi-model agent that calls external tools has a large attack surface. Prompt injection through user-controlled content can redirect a tool-calling model to perform unintended actions. Validate all tool inputs. Scope each model call to the minimum context it needs. Do not include credentials or sensitive system information in the context of the tool-calling model if they are not needed for that call.

Frequently Asked Questions

Which LLM is best for building an AI agent?

There is no single best model for an agent because agents have multiple roles. For routing and classification, use Haiku 4.5. For tool-calling and structured output, use Sonnet 4.6. For complex multi-step reasoning, use Opus 4.8. Matching the model to the role reduces cost by 60 to 80 percent while maintaining or improving output quality on the tasks that matter.

Should I use GPT-4 or Claude for my AI agent?

For tool-calling agents with strict schema requirements, Claude Sonnet 4.6 with strict: true on tool definitions gives you schema-validated inputs out of the box, which reduces the error rate on downstream API calls. For long-horizon autonomous tasks, Claude Opus 4.8 has a 1M-token context window and adaptive thinking that adjusts reasoning depth to the task. The right answer depends on your workload; run evals on both before committing.

How do I reduce the cost of my AI agent without sacrificing quality?

Three levers: first, route simple classification and extraction tasks to a cheap model like Haiku instead of a frontier model. Second, enable prompt caching on stable context blocks (system prompt, tool schemas, reference documents) to reduce effective input token cost by up to 90 percent on cache hits. Third, use retrieval to give the model a clean, scoped context instead of a large noisy context, which often lets you use a cheaper model tier for the same task quality.

What is the difference between a router, a tool-caller, and a reasoner in an AI agent?

The router classifies input and decides which path to take. It is a simple classification task suited to a cheap, fast model. The tool-caller decides which external functions to invoke, formats the calls correctly, and integrates results. It requires reliable schema following, not deep reasoning. The reasoner handles open-ended tasks requiring judgment, planning, or synthesis across multiple sources. Only the reasoner benefits materially from a frontier model tier.

How do I evaluate which LLM to use for my specific use case?

Build a labeled test set of 50 to 200 representative inputs for each role in your agent. Run each candidate model on the full set with your actual prompts. Score outputs against your acceptance criteria: accuracy for classification, schema validity for tool-calling, quality rubric for final responses. Compare cost per correct output, not cost per token. The model that maximizes correct outputs per dollar for each role is the right choice, regardless of benchmark rankings.

Does using a smaller model for routing hurt agent quality?

Only if your routing taxonomy is poorly defined or your prompts are vague. A tight routing prompt with clear category definitions and worked examples runs reliably on Haiku with greater than 95 percent accuracy on most commercial tasks. The routing failure mode is almost always prompt design, not model capability. If a smaller model is routing incorrectly, fix the prompt before upgrading the model tier.

Build an Agent That Is Correct, Not Just Impressive

The agents that work in production are the ones designed with explicit decisions at every layer: which model handles which role, what context each model sees, where validation happens, and how failures surface. That design work is not glamorous, but it is the difference between a demo and a deployed system. If you are building a production AI agent and want an experienced architect who has done this before, I can help at every stage from architecture to deployment. Start with my AI Agent Development service page, or reach out directly at /contact. Let me help you build an AI agent that works in production.