How to Test an AI Agent With Evals
You test an AI agent with evals: a repeatable suite that runs the agent against a fixed set of inputs and scores each output against what you expected. The setup has four parts. First, build a golden dataset of real inputs paired with expected outputs or pass conditions. Second, run the agent over that dataset. Third, score each result, using exact match where the answer is deterministic, and an LLM-as-judge (a second model grading the output against a rubric) where the answer is open-ended. Fourth, gate your deploys on the score so no prompt, model, or tool change ships without passing. Traditional unit tests assume the same input gives the same output; a model does not, so evals measure quality as a distribution, not a single pass or fail.
I'm Mahmoud Zalt, an AI systems architect. Through Sista AI I help teams put evals and guardrails around agents before they reach users, and this is the setup I put in place first.
Why Agents Need Evals, Not Just Unit Tests
A unit test asserts that a function returns exactly what you expect. That model breaks with agents for two reasons. The output is non-deterministic: run the same prompt twice and the wording changes, so a string-equality assertion is useless. And the output is open-ended: 'summarize this ticket' has no single correct answer, only better and worse ones. Evals handle both by scoring quality across many examples and tracking how that quality moves over time. The practical payoff is regression detection. Model providers update models, you tweak a prompt, someone changes a chunking parameter, and any of those can quietly degrade behavior on inputs you were not looking at. An eval suite catches the regression before your users do, which is the entire point of testing.
Building the Eval Suite Step by Step
Here is the sequence I use to stand up evals on an agent that has none:
- Collect a golden dataset. Pull twenty to a hundred real inputs, ideally from actual usage or support tickets, and label each with an expected output or a clear pass condition. Cover your common cases and your known edge cases.
- Pick a scoring method per case. Use exact or structural match for deterministic outputs (a JSON field, a classification label), and LLM-as-judge with a written rubric for open-ended ones. For retrieval-heavy agents, also score whether the retrieved context actually contained the answer.
- Write the judge rubric carefully. A vague rubric gives noisy scores. Spell out what a good answer must contain and what disqualifies it, and prefer a small scale (pass or fail, or one to five) over false precision.
- Run and baseline. Run the suite, record the score, and treat that number as the line no change is allowed to cross downward without a decision.
- Gate deploys. Wire the suite into CI so a prompt, model, or tool change runs the evals and blocks on a regression.
Start small. A tight fifty-example suite that runs on every change beats a thousand-example suite that runs once and rots.
What to Measure, and the Traps to Avoid
Measure the dimensions that map to how your agent actually fails. Common ones: task success (did it accomplish the goal), factual grounding (is every claim supported by retrieved context), format correctness (valid structured output), tool-use accuracy (did it pick and call the right tool), and safety (did it refuse what it should refuse). For retrieval specifically, separate retrieval quality from answer quality, because an agent can answer correctly on lucky retrieval, or retrieve perfectly and still answer poorly, and you want to know which one broke.
Frequently Asked Questions
What is an eval in AI agents?
An eval is an automated test that runs an agent against a fixed dataset of inputs and scores each output against an expected result or rubric. Because model outputs vary, evals measure quality across many examples rather than asserting a single exact answer like a unit test.
What is LLM-as-judge?
LLM-as-judge uses a second model to grade an agent's output against a written rubric. It scales evaluation of open-ended answers that have no single correct string. It works well when the rubric is specific, but you should spot-check its scores by hand because judges can carry biases.
How many examples does a good eval set need?
Start with twenty to a hundred well-chosen examples covering common and edge cases. Quality and coverage matter far more than volume. A small suite that runs on every change catches more regressions than a large one that runs rarely.
Can I test agents with normal unit tests?
Unit tests still cover the deterministic parts, your tool functions, schema validation, and parsing. But the model's own behavior is non-deterministic and open-ended, so it needs evals. Use both: unit tests for the plumbing, evals for the model's output quality.
Set Up Evals on Your Own Agent
Evals are the difference between shipping agent changes on faith and shipping them on evidence. Once a suite gates your deploys, you can upgrade models, rewrite prompts, and refactor retrieval without holding your breath, because a regression shows up as a failing score instead of an angry user.
If you want to build an eval harness on your real agent rather than a sample, that is one of the things we do in my hands-on AI Agents for Engineers masterclass. It is private, one-on-one or with your own team, covering agent architecture, tools and function calling, memory and retrieval, orchestration, and evals and guardrails. It starts at $120 for a single private technical session, $420 for a four-session Engineering track, or $780 for a private team workshop.
