Observability and Evals for AI Agents, Explained
Observability and evals are the two halves of knowing whether your agent actually works in production. Observability is about seeing: it traces every step the agent takes, the prompts, the model responses, each tool call and its result, plus tokens, latency, and cost, so that when something goes wrong you can replay exactly what happened. Evals are about measuring: they score the quality of the agent's outputs against a dataset of examples and criteria, so 'is it good?' becomes a number you can track instead of a feeling. Observability tells you what the agent did on a given run; evals tell you how well it does across many runs. You need both, because an agent is non-deterministic: the same input can produce different outputs, so you cannot verify it by eyeballing a few examples the way you would a normal function.
I'm Mahmoud Zalt, an AI systems architect. At Sista AI I help teams measure and observe their agents so quality stops being a guess.
Observability: See Every Step
An agent run is a chain of decisions, and without tracing it is a black box. Observability opens the box by recording the run as a trace made of spans, one span per step, nested to show what called what. For agents, a useful trace captures at each step:
- The exact prompt sent to the model, including the retrieved context and tool definitions, not just the user's message.
- The raw model response, including any tool-call requests it made.
- Every tool call: the arguments, the result, and whether it succeeded or failed.
- Cost and performance: tokens in and out, latency, and dollar cost per step and per run.
Why this matters: when an agent gives a bad answer, the cause is almost never mysterious once you can see the trace. Usually retrieval fetched the wrong context, a tool returned an error the model then improvised around, or the prompt ballooned past the point where the model could focus. Traces turn 'the agent is flaky' into a specific, fixable step. They are also where you watch cost, since a single misbehaving loop can quietly multiply token spend.
Evals: Measure Quality on Purpose
Evals are how you replace 'seems fine' with evidence. The core idea is a test suite for a non-deterministic system: a dataset of representative inputs, plus a way to judge whether each output is good. They come in two modes.
- Offline evals run against a fixed dataset before you ship a change. You assemble examples, including the tricky and past-failure cases, and score new prompts, models, or logic against them. This is your regression net.
- Online evals run against real production traffic. You sample live runs and score them to catch drift and failure modes your dataset never imagined.
How you score depends on the task. Some outputs have a correct answer you can check directly. Many do not, and there you use graded criteria: rubrics a reviewer applies, or an LLM-as-judge, a separate model prompted to score outputs against a clear rubric. LLM-as-judge scales human judgment but is itself imperfect, so you calibrate it against human ratings on a sample rather than trusting it blindly. The discipline that makes evals worth the effort is boring but decisive: every real bug becomes a new eval case, so the same failure can never quietly return.
How They Work Together in Production
Observability and evals form a loop that keeps an agent honest over time. In practice it runs like this:
- Observe production runs and flag the ones that look wrong or expensive.
- Diagnose each with its trace to find the actual failing step.
- Capture that failure as a new eval case with the expected outcome.
- Change the prompt, retrieval, tools, or model to fix it.
- Re-run the eval suite to confirm the fix and prove nothing else regressed.
This loop is what makes iterating on an agent safe. Without it, every prompt tweak is a gamble: you fix one case and silently break three others, because you have no way to see the collateral damage. With it, quality only ratchets upward.
Frequently Asked Questions
What is the difference between observability and evals for AI agents?
Observability is seeing what an agent did on a single run by tracing its steps, tools, tokens, and cost. Evals are measuring how well it performs across many runs by scoring outputs against a dataset. One diagnoses individual runs; the other quantifies overall quality.
How do you evaluate an AI agent that has no single correct answer?
You score against graded criteria instead of exact matches. That means human review with a rubric, or an LLM-as-judge model prompted to grade outputs against clear criteria. Calibrate the judge against human ratings on a sample so you trust its scores.
What is LLM-as-judge?
It is using a separate language model to score another model's outputs against a rubric. It scales evaluation far beyond what humans can review by hand, but it is imperfect, so you validate it against human judgment rather than treating its scores as ground truth.
When should I add observability and evals?
Before you spend serious time tuning, not after. Without traces you are debugging blind, and without an eval set every change is a gamble that may fix one case and break others. Even a small dataset and basic tracing make iteration measurable from day one.
Make Agent Quality Something You Can Measure
Observability and evals are what turn an agent from a demo you hope works into a system you can improve with confidence. See every step, score every change against real examples, and feed each production failure back into the suite. That loop is the difference between guessing and engineering.
If you want to stand up tracing and an eval suite on your own agent, choosing what to trace, building the dataset, and wiring LLM-as-judge sensibly, that is hands-on ground in my AI Agents for Engineers masterclass, alongside agent architecture, tools and function calling, memory and retrieval, orchestration, and guardrails. It is always private, one-on-one or with your team, from $120 for a single technical session, $420 for a four-session Engineering track, or $780 for a private team workshop.
Instrument and evaluate your agent in the AI Agents for Engineers masterclass
