What Building an Agent Actually Means in 2026

An agent is a system where an LLM controls the flow of execution: it picks which tool to call, in what order, and decides when to stop. That is different from a workflow, where the steps are hardcoded by a human and the LLM only fills in slots. The interesting part of agent building is everything that surrounds the LLM: the tool belt it can reach, the memory it carries between steps, the supervisor that catches it when it loops, the evaluation harness that tells you whether a code change made it better or worse, and the cost and latency budgets that decide how long it is allowed to think.

LangChain reported in their 2026 State of Agent Engineering that over 60% of agent production incidents come from state management failures, not model failures. That number tracks what I see in practice. Most agents that fail in production fail because their memory is wrong, their checkpoints leak, their tool errors are unhelpful, or their loop has no cost ceiling, not because the underlying model is bad.

• Anthropic distinction: workflow uses predefined code paths, agent uses LLM to dynamically direct tool use and flow
• The minimal loop is plan, act, observe, repeat until done or budget exhausted
• Tool use is the actual primitive: function calling plus MCP servers plus structured outputs
• Memory splits into short-term (context window, scratchpad) and long-term (vector, key-value, graph)
• Observability is mandatory from day one: full trajectory traces, not just final outputs
• Cost shape: an agent run is typically 5x to 100x a single LLM call, budget accordingly
• Senior work happens before framework choice, not in framework choice

Picking the Right Topology

Most teams reach for multi-agent before they have a working single-agent. Cognition Labs published a widely read essay arguing single-agent systems with strong context engineering beat multi-agent setups for most coding workloads, because context fragmentation produces incoherent results. Anthropic published the opposite case for research agents: their lead-researcher-with-subagents pattern scored 90% better on research evals, at 15x the token cost. Both are right. The topology choice depends on the task shape, not the trend.

• Single agent: one loop, one context, simplest to debug, default starting point
• Sequential / prompt chaining: hardcoded steps, LLM only at each step, cheapest and most reliable
• Routing: classifier LLM picks one downstream path, lighter than full agent loops
• Supervisor (hub-and-spoke): planner agent delegates atomic tasks to specialist workers, aggregates results
• Network / swarm handoffs: peer agents transfer control, used by OpenAI Agents SDK
• Parallel orchestrator-workers: planner spawns N workers, results merged, powers deep research patterns
• Evaluator-optimizer loop: one agent generates, another critiques, iterates to threshold
• Human-in-the-loop checkpoint: graph pauses for approval on irreversible actions
• Rule of thumb: start single-agent, escalate only when evals show the topology is the bottleneck

Tools and the MCP Layer

The single highest-leverage decision in agent building is the tool surface. A well-shaped tool belt makes a mediocre model behave reasonably. A badly-shaped one makes a top-tier model loop forever. Function calling is the baseline API surface. Model Context Protocol (MCP) is the standard that lets one tool implementation serve every modern client. Most production agents I build now consume tools via MCP and expose their own sub-agents as tools to a supervisor.

• Provider-native function calling is the baseline; schemas should be tight, names verb-like, descriptions example-rich
• MCP standardizes tool/resource/prompt exposure across Claude, Cursor, ChatGPT, Windsurf, Claude Code
• Keep tool count low per agent: above 30 tools in one context, selection accuracy degrades measurably
• Return structured, machine-parseable results, not free-form prose
• Rich error messages: 404 user not found, try search_users beats a stack trace every time
• Idempotency keys on every write tool to survive retries without duplicate side effects
• Confirmation gates on destructive tools: deletes, sends, charges, never just trust the model
• Sub-agents as tools is the cleanest hierarchical pattern

Memory Architecture

Memory in agents has multiple layers and no consensus on which to use when. Short-term memory is the context window plus any scratchpad inside it. Long-term memory persists across runs and is implemented as vector stores for semantic recall, key-value stores for facts, or graph stores (Zep, Mem0, Graphiti, Letta) for entity relationships. The hard problem is not storage, it is retrieval policy: when does the agent ask its memory, and what does it ask for.

• Short-term: prompt window, scratchpad, current trajectory, governed by context engineering
• Episodic long-term: past sessions, summarized then embedded, retrieved by similarity
• Semantic long-term: facts, preferences, user model, often key-value or graph
• Procedural memory: learned tool-use patterns, sometimes stored as few-shot exemplars
• Shared state across agents: a typed object (LangGraph state) or message bus (AutoGen)
• Compaction: summary buffers, hierarchical summarization, attention-sink eviction
• Frameworks worth naming: LangMem, Mem0, Zep, Letta (formerly MemGPT), each with different write/retrieve policies

Frameworks I Use and When

Framework choice is downstream of architecture. The architecture comes first. That said, framework choice has measurable impact: independent benchmarks in 2026 show framework choice can shift agent benchmark performance by up to 30 percentage points on identical models. Picking the wrong one costs real accuracy.

• LangGraph: production standard for stateful, auditable workflows, durable checkpointing, time-travel debugging, human-in-the-loop nodes. My default for non-trivial agents
• OpenAI Agents SDK: handoff-first, lightweight, fast to ship for OpenAI-native stacks. Released March 2025, replaced Swarm
• Microsoft Agent Framework: Azure-native, strong for enterprise Azure deployments
• CrewAI: fastest path to a working multi-agent prototype, role-and-task abstraction, weaker production controls. Good for evaluation spikes
• AutoGen: conversational multi-agent, GroupChat patterns, strong for research and exploratory flows
• Pydantic AI: type-safe, Python-native, single-agent strong, great when team already lives in Pydantic
• Mastra: TypeScript-native agent framework, good fit when full stack is TS and team avoids Python
• Bare-metal Anthropic or OpenAI SDK: when the use case is simple enough that a framework is overhead

Evaluation and Observability

You do not have an agent until you have an evaluation harness. The harness is what tells you whether last week edit made things better, worse, or sideways. Output-only evals are not enough for agents because the same final answer can be reached by good and bad trajectories. Trajectory evals measure the path, including tool-choice correctness and step efficiency, not just the answer.

• Trajectory evals: score path quality, tool selection, step efficiency, and final correctness independently
• LLM-as-judge with explicit rubrics, calibrated against human labels on a holdout set
• Golden trajectories: pinned known-good runs, alert on divergence
• Regression suite that runs on every prompt or model change before deploy
• Observability stack: LangSmith for LangChain, Langfuse open source, Braintrust, Arize Phoenix, MLflow LLM tracing
• Token, latency, and cost dashboards per node, per tool, per tenant
• Failure mode catalog: context rot, tool overload, planning drift, sub-agent incoherence, irrecoverable side effects

Durability and Recovery

Agents fail constantly: tools error, models hallucinate arguments, plans diverge, timeouts hit. Production agents need the same disciplines as distributed systems. The cardinal sin is unbounded loops. Every agent gets a hard step cap and a cost cap enforced by the runtime, not by the model.

• Checkpointing: persist state after every node, runs are resumable, time-travel debugging is possible
• Idempotency on every write so retries do not duplicate side effects
• Retry policies with exponential backoff and jitter, terminal-vs-retryable error distinction
• Fallbacks: secondary model, smaller model, canned response when primary fails
• Hard budgets: max steps, max tokens, max wall time, max dollars, enforced outside the model
• Human-in-the-loop gates on destructive or irreversible actions
• Per-run isolation: one bad input does not poison the next run
• Dead-letter queue and replay tooling for failed runs

When NOT to Build an Agent

The honest answer to should I use an agent is usually no. If the task shape is fixed, a workflow is cheaper, faster, easier to test, and easier to operate. Agents earn their cost only when the branching is irreducible and the cost of getting it wrong is bounded. Most teams should ship a workflow first, instrument it, and only graduate to an agent when the eval results clearly require it.

• Task has fixed shape: write a workflow with one LLM step per node
• Latency must be sub-second: agent loops do not fit
• Action space is small: a router plus a couple of fixed tools is simpler
• Cost of error is unbounded: do not let an agent take irreversible actions without a hard gate
• Token budget is tight: agent runs are 5x-100x a single LLM call
• Team has never shipped an LLM feature: start smaller, build the eval muscle, then graduate
• When in doubt: simplest LLM call first, then chain, then router, then agent

What I Ship in a Typical Engagement

A typical agent build with me runs four to twelve weeks. The shape depends on whether the agent is internal-facing (operations, knowledge work) or user-facing (in your product). Internal agents I usually ship myself end-to-end. User-facing agents I usually ship in close partnership with your team because the product surface needs your domain knowledge.

• Week 1: discovery, task taxonomy, target metric definition, tool inventory, topology decision
• Week 2: evaluation harness, golden trajectories, baseline metrics on a non-agent solution
• Week 3-6: build, instrument, iterate against the eval harness
• Week 7-8: hardening, durability tests, cost and latency tuning, on-call runbook
• Week 9+: pilot rollout, observability dashboards, optional advisory tail
• Deliverable: the agent, the eval harness, the observability dashboards, the runbook, the handoff doc
• I write code your engineers can extend, with tests and design docs, not a black box

FAQ