What Agentic Architecture Actually Means

An agent is a system where an LLM controls the flow of execution: it chooses which tool to call, in what order, and when to stop. That contrasts with a workflow, where a human author hardcodes the steps and the LLM only fills in the blanks. Agentic architecture is the set of structural decisions, model role, tool surface, memory layout, control loop, and recovery policy, that make this controllable in production.

• Anthropic's distinction: workflow = predefined code paths with LLM steps; agent = LLM dynamically directs tool use and flow
• The minimal agent loop is plan, act, observe, repeat until done or budget exhausted
• Agentic value scales with task open-endedness, irreducible branching, and the cost of writing explicit workflow logic
• Latency, token cost, and failure surface all grow super-linearly with agent depth
• Most "agent" demos in production are actually augmented LLM calls or prompt chains, not true agents
• The first architectural question is always: does this task need an agent, or a pipeline with one LLM step?

Single-Agent, Multi-Agent, and Hierarchical Patterns

Single-agent designs put one model in a loop with a tool belt. They are easier to debug, evaluate, and cache, which is why Cognition Labs has publicly argued (in "Don't Build Multi-Agents") that single-agent systems with strong context engineering outperform multi-agent setups for most coding workloads. Multi-agent designs split work across specialist agents, useful when tasks have genuinely parallel subgoals.

• Single-agent: one loop, one context, simplest mental model, used by Cursor chat and most Claude Code flows
• Multi-agent peer: agents with equal authority pass messages, e.g. AutoGen GroupChat or CrewAI sequential
• Hierarchical supervisor: a planner agent delegates to workers, e.g. LangGraph supervisor, Anthropic research subagents
• Network/swarm: any agent can hand off to any other, OpenAI Swarm/Agents SDK style, good for routing but hard to reason about globally
• Anthropic's multi-agent research system used a lead researcher spawning parallel subagents; their writeup credits a 90% improvement on research evals but warns about token overhead (15x a chat turn)
• Pick multi-agent only when tasks parallelize cleanly or specialization beats context-sharing cost

Orchestration Patterns

Orchestration is the topology of who talks to whom and who decides what runs next. Dominant patterns: sequential (prompt chaining), supervisor/hub-and-spoke (central router), swarm/network (peer handoffs), and parallel fan-out with aggregation. LangGraph models all as a directed graph with explicit state; OpenAI's Agents SDK uses handoffs; AutoGen uses conversation; CrewAI uses processes.

• Sequential / prompt chaining: deterministic step order, best when subtasks are well-defined and serial
• Routing: a classifier LLM picks one downstream path, simpler and cheaper than full agent loops
• Supervisor (hub-and-spoke): central agent owns plan, delegates atomic tasks, aggregates results
• Swarm / network handoffs: peer-to-peer transfers of control, used by OpenAI Swarm and Agents SDK
• Parallel orchestrator-workers: planner spawns N workers, results merged; powers most "deep research" features
• Evaluator-optimizer loop: one agent generates, another critiques, repeats until threshold met (Reflexion-style)
• Human-in-the-loop checkpoint: graph pauses for approval, native in LangGraph and Anthropic tool-use flows

Tool Use, Function Calling, and MCP

Tools are how agents touch the world. Modern model APIs expose structured function calling. The Model Context Protocol (MCP), open-sourced by Anthropic in late 2024, standardizes how tools, resources, and prompts are exposed to any agent. Good tool design is the highest-leverage agent work.

• Provider-native function calling is the baseline; schemas should be tight, names verb-like, descriptions example-rich
• MCP standardizes tools/resources/prompts across servers; one MCP server can serve Claude, Cursor, ChatGPT, etc
• Keep tool count low per agent; >30 tools in a single context degrades selection accuracy measurably
• Return rich, structured errors - the agent recovers far better from "404 user not found, try search_users" than from a stack trace
• Idempotency keys on write tools prevent duplicate side effects when the agent retries
• Wrap dangerous tools (delete, send, charge) in confirmation gates or dry-run modes
• Sub-agents themselves can be exposed as tools, a clean pattern for hierarchical systems

Memory Architecture

Memory in agents splits into short-term (the working context window, including scratchpads) and long-term (anything persisted across runs). Long-term memory is usually implemented as vector stores for semantic recall, key-value stores for facts, or graph stores (Zep, Mem0, Graphiti) for entity relationships. The hard problem is not storage; it is retrieval.

• Short-term: the prompt window, scratchpad, and current tool trajectory, governed by context engineering
• Episodic long-term: past sessions, often summarized then embedded, retrieved by similarity
• Semantic long-term: facts, preferences, user model, often key-value or graph
• Procedural memory: learned tool-use patterns, sometimes stored as few-shot exemplars
• Shared state: a typed object (LangGraph) or shared message bus (AutoGen) all agents read and write
• Compaction strategies: summary buffers, hierarchical summarization, attention-sink eviction
• Frameworks worth naming: LangMem, Mem0, Zep, Letta (formerly MemGPT), all encode different write/retrieve policies

Planning Patterns

ReAct interleaves reasoning traces with actions and remains the default for most tool-using agents. Plan-and-Execute separates a planner from an executor, reducing token cost on long tasks. Reflexion adds verbal self-critique. Tree-of-Thoughts explores multiple branches with backtracking. In production, hybrids dominate.

• ReAct: thought, action, observation loop, the default and the right starting point
• Plan-and-Execute: plan once, execute many, cheaper and more controllable for long horizons
• Reflexion: after a failure or attempt, generate a verbal critique and store it as a lesson
• Tree-of-Thoughts: explore N branches, evaluate, prune, backtrack, expensive but strong on puzzles
• Graph-of-Thoughts and Skeleton-of-Thought: variants for non-linear and parallelizable reasoning
• Self-consistency: sample N trajectories, take the majority, cheap reliability boost
• Anthropic extended thinking and OpenAI reasoning models absorb some planning into the model itself, simplifying the outer loop

Recovery, Durability, and Operational Discipline

Agents fail constantly: tools error, models hallucinate arguments, plans diverge, timeouts hit. Production agents need the same disciplines as distributed systems. The cardinal sin is unbounded loops: every agent should have a step cap and a cost cap enforced by the runtime, not by the model.

• Checkpointing: persist state after every node so runs are resumable, LangGraph checkpointer or your own
• Idempotency: every write tool needs a key so retries do not duplicate
• Retry policies: exponential backoff with jitter, distinguish retryable vs terminal errors
• Fallbacks: secondary model, smaller model, or canned response when primary fails
• Budgets: hard caps on steps, tokens, dollars, and wall time, enforced outside the model
• Human-in-the-loop gates: pause for approval on irreversible actions
• Observability: full trajectory logs, not just final outputs - LangSmith, Langfuse, Braintrust, Arize Phoenix

Evaluation, Failure Modes, and When NOT to Use Agents

Agents demand trajectory-level evaluation, not just output evaluation. The honest answer to "should I use an agent" is usually no. If the task has a fixed shape, a workflow is cheaper, faster, and easier to test. Use agents where the branching is irreducible and the cost of getting it wrong is bounded.

• Trajectory evals: score the path, not just the answer, includes tool-choice accuracy and step efficiency
• LLM-as-judge with explicit rubrics, calibrated against human labels on a holdout set
• Golden trajectories: pin known-good runs, alert on divergence
• Failure modes: context rot, tool overload, planning drift, sub-agent incoherence, irrecoverable side effects
• Cost shape: agents are 10-100x the tokens of a single call, budget accordingly
• Skip the agent when: the workflow is fixed, latency must be sub-second, the action space is small, or the cost of error is unbounded
• Start with the simplest thing (single LLM call, then chain, then router, then agent) and only escalate when evals demand it

FAQ