What Are AI Evals and Why They Matter More Than Your Prompt

Evals Are the Test Suite for Non-Deterministic Systems

An eval is a structured test that measures whether your AI system produces correct, useful, and safe outputs across a defined set of inputs. Without evals, you are not running a product, you are running a vibe check. Every prompt change, model upgrade, retrieval tweak, or guardrail addition you make without an eval suite is a deployment into the dark.

I am Mahmoud Zalt, a senior AI systems architect with 16+ years building production software since 2010. I founded Sista AI, where keeping a workforce of autonomous agents reliable in production for the past year has made evals a daily discipline rather than an afterthought. I design and ship LLM-powered systems end to end, from retrieval and tool-calling to observability and safety layers. If you want deep technical guidance on AI architecture, my AI agent development services are where that work happens. You can also read more about my background or see past projects.

Why Your Prompt Is Not a Quality Signal

A prompt is an instruction. An eval is evidence. Teams that invest weeks in prompt engineering but zero hours in evals are optimizing blindly. Here is what happens in practice: you tweak a system prompt to fix a hallucination on case A. It works. You ship. Two weeks later, a regression appears on case B that you never tested. You tweaked the prompt again. The cycle continues with no visibility into whether you are net positive or net negative over time.

LLMs are non-deterministic. The same prompt at temperature 0.7 will produce different outputs across calls. Model providers silently update weights. Context windows change what the model attends to. RAG retrieval quality shifts as your corpus grows. None of these changes are visible through manual spot-checking. They are only visible through a systematic eval suite that you run on every change, the same way you run unit tests before merging code.

The analogy is exact: you would not ship a backend API without tests because the function is deterministic and you trust it. LLMs are not deterministic and you should not trust them without tests. Evals are the mechanism that lets you earn that trust incrementally.

The Three Types of Evals You Actually Need

Most teams jump straight to LLM-as-judge without understanding the full stack. Here is the practical breakdown:

1. Exact-Match and Rule-Based Evals

The fastest and cheapest. You define expected output or constraints and assert deterministically. Examples: does the output contain a valid JSON schema, does the response stay under 200 words, does the classification output one of the allowed labels, does the SQL query not contain a DROP statement. These run in milliseconds, cost nothing, and catch a large class of regressions immediately. Start here. Most teams skip this and go straight to LLM-as-judge, which is a mistake.

2. Golden Dataset Evals

You curate a set of input-output pairs where you have ground truth, either human-labeled or sampled from production logs you manually reviewed. The model's output is compared against the golden answer using a scorer: exact string match, BLEU/ROUGE for summarization, semantic similarity (cosine on embeddings), or structured field extraction accuracy. Golden datasets are expensive to build but irreplaceable for regression tracking. A dataset of 50 to 200 well-curated examples is worth more than a thousand auto-generated ones. Invest in curation.

3. LLM-as-Judge Evals

You use a second LLM (usually a stronger model like Claude Opus or GPT-4o) to score the outputs of your production model on dimensions like correctness, helpfulness, grounding (does the answer stay within the retrieved context), and safety. This scales to open-ended tasks where exact-match is impossible. The tradeoff: it costs money, it introduces a second point of model failure, and judge models have their own biases. Calibrate your judge against human labels before trusting it. If your judge agrees with human raters less than 85% of the time, it is not a reliable signal.

Building a Golden Dataset That Actually Holds Up

A golden dataset is a curated collection of representative inputs paired with correct, expected outputs. It is your ground truth. Here is how I build one in production:

1. Seed from real traffic. Sample 200 to 500 inputs from production logs in the first week. These are the inputs your users actually send, not the ones you imagine. Filter for diversity across intent categories, edge cases, and failure modes you already know about.
2. Label carefully. For each input, write the ideal output (or the acceptance criteria for outputs you cannot enumerate exactly). This is human work. Do not auto-generate labels with the same model you are evaluating. That is circular. Use a stronger model as a labeling assist, then human-review every example before it enters the golden set.
3. Version your dataset. Store it in git alongside your prompts and code. Every eval run references a specific dataset version. When you add new failure cases (post-incident), you add them to the dataset and bump the version. This is how you track regression coverage over time.
4. Keep it small and high-quality. 50 to 200 examples, all human-reviewed, beats 5000 auto-generated examples. Quality of signal matters more than volume. A noisy golden set gives you false confidence, which is worse than no evals at all.
5. Split by category. Segment by intent or feature. If you have a customer support bot, split into billing questions, technical issues, refund requests, and out-of-scope queries. This lets you see regressions in specific categories rather than just an aggregate score that masks failures.

How to Use LLM-as-Judge Without Getting Burned

LLM-as-judge is powerful and widely misused. Here is what teams get wrong and how to do it properly.

What Teams Get Wrong

The most common mistake is writing a single vague judge prompt like 'Rate this response on a scale of 1 to 10 for quality.' This produces inconsistent scores, does not tell you what is wrong, and cannot be calibrated. The second mistake is using the same model family as both producer and judge, which creates a blind spot where both models share the same failure modes. A GPT-4o judge will be more forgiving of GPT-4o outputs than a Claude judge will be.

How to Build a Reliable Judge

Decompose quality into specific, measurable dimensions. For a RAG system, the dimensions are: grounding (is every claim traceable to the retrieved context), completeness (does the answer address all parts of the question), conciseness (no unnecessary padding), and safety (no harmful content). Write a separate judge prompt for each dimension. Score each 1 to 3 (not 1 to 10, coarser scales are more consistent). Log raw scores plus the judge's reasoning chain for debugging.

Then calibrate. Take 50 examples and score them with both your judge and a human rater. Compute agreement rate (Cohen's kappa or simple percent agreement). If agreement is below 80%, your judge prompt needs revision. This calibration step is non-optional if you want to trust your eval pipeline.

Short Worked Example: Grounding Eval

Judge prompt (grounding dimension):
You are evaluating whether an AI answer is grounded in the provided context.
Context: {retrieved_chunks}
Question: {user_question}
Answer: {model_output}

Score 1-3:
3 = Every claim in the answer is directly supported by the context.
2 = Most claims are supported; one minor inference not in context.
1 = One or more claims contradict or go beyond the context.

Output JSON: {'score': N, 'reason': '...'}

Note the structured JSON output requirement. This makes scores programmatically parseable and prevents the judge from padding its response.

Regression Tracking: Treating AI Changes Like Code Changes

A one-time eval run is a benchmark. A regression tracking system is an eval suite that runs automatically on every change and alerts you when scores drop. This is the gap between teams that ship AI products with confidence and teams that are perpetually surprised.

Here is the minimal setup I use:

• Eval runner in CI. On every pull request that touches a prompt, retrieval config, model version, or agent tool definition, the eval suite runs automatically. The PR is blocked if any category score drops more than 5 percentage points from the baseline. This is the same discipline as blocking a PR that breaks unit tests.
• Baseline pinning. After each intentional improvement, you accept the new scores as the baseline. The system tracks the delta from baseline, not an absolute threshold. This prevents alert fatigue from noisy dimensions while still catching real regressions.
• Separate evals per component. In an agent system, you evaluate the retrieval layer (retrieval recall@K), the generation layer (groundedness, completeness), the tool-calling layer (correct tool selected, correct arguments), and the end-to-end flow (task completion rate) independently. A drop in retrieval recall that does not affect end-to-end task completion is a warning. A drop in end-to-end completion rate is a block.
• Observability integration. Log every eval run with structured metadata: prompt version, model version, dataset version, timestamp, per-category scores. Store in a time-series view so you can see score trends across weeks. Tools like LangSmith, Braintrust, and Arize support this natively. If you are self-hosting, a Postgres table with a Grafana dashboard is sufficient.

Evals for Agents and Tool-Calling Systems

Standard text-quality evals are not sufficient for agentic systems. When your LLM is selecting tools, calling APIs, reading from memory, and taking multi-step actions, you need a different evaluation model because the failure surface is wider and the consequences of failure are higher.

For tool-calling and MCP-based agents, I evaluate at three levels:

Tool Selection Accuracy

Given a user request, did the agent select the correct tool (or tools)? This is a classification problem with a known correct answer. Build a golden set of 50 to 100 requests, label the expected tool call, and measure precision and recall. A well-tuned agent should hit above 90% on this. Below 85% means your tool descriptions are ambiguous or your routing logic is wrong.

Argument Extraction Accuracy

Even when the right tool is selected, the arguments may be wrong. Evaluate whether the agent correctly extracted the required parameters from the user's natural language input. Test edge cases: missing required fields, ambiguous values, type coercion errors. This is where most agent bugs live in production.

Task Completion Rate

End-to-end: given a task, did the agent complete it successfully as measured by a verifiable outcome? For a booking agent, did a booking actually get created in the database? For a code-generation agent, does the generated code pass the provided test suite? This is the most meaningful metric and the hardest to automate. Define completion criteria before building, not after.

Human-in-the-loop review is also an eval. For high-stakes actions (sending emails, making purchases, deleting records), instrument your system so a human approval gate can sample and review a percentage of actions. Track approval rates and override rates. A rising override rate is an early signal of model drift or prompt regression.

How to Prioritize Evals When You Have Limited Time

You cannot build every eval on day one. Here is the order I follow for a production AI system:

1. Safety and format evals first, rule-based, zero cost. Does the output stay within the allowed output schema? Does it avoid prohibited content categories? These run on every request in production as a guardrail, not just in CI.
2. Golden dataset on your top 3 use cases. Identify the three highest-volume or highest-stakes intents. Build 20 to 30 golden examples for each. Run these in CI. This is usually two to three days of work and gives you 80% of the regression protection you need.
3. LLM-as-judge for open-ended dimensions once you have calibrated it. Add grounding eval if you have RAG. Add helpfulness eval if you have a conversational assistant. Do not add judge dimensions you cannot calibrate.
4. Tool-calling accuracy evals before you expand your agent's tool catalog. Every new tool you add should come with 10 to 20 golden examples that test its selection and argument extraction.
5. End-to-end task completion evals for your most critical flows. These are expensive to build but they are the only signal that tells you whether your system actually does what it is supposed to do.

A practical cost note: running a 200-example golden dataset with Claude Haiku as judge costs under $1 per run. There is no budget justification for skipping evals. The cost of a production regression, in user trust, support tickets, and engineering time, is orders of magnitude higher.

Frequently Asked Questions About AI Evals

What are evals in AI and how are they different from tests?

Evals are structured measurements of AI system output quality. Traditional software tests check deterministic behavior: given input X, assert output Y. Evals measure probabilistic quality: given a distribution of inputs, measure correctness, grounding, safety, and task completion rates across a defined dataset. The underlying principle is the same (systematic quality verification before shipping) but the implementation differs because LLM outputs are not deterministic and cannot be compared with simple equality checks.

Do I need evals if I am just using the OpenAI API with a simple prompt?

Yes. Especially then. Simple prompts are the most fragile AI systems because they have no retrieval, no guardrails, and no structured output parsing to catch errors. When OpenAI silently updates a model version, your prompt's behavior changes. When your use case evolves and you tweak the prompt, you have no way to know if you introduced a regression. A set of 30 to 50 golden examples with rule-based and LLM-as-judge scoring takes one day to build and immediately gives you regression protection.

What is LLM-as-judge and is it reliable?

LLM-as-judge means using a second language model to score the outputs of your production model on specific quality dimensions. It is reliable when calibrated: you validate that the judge's scores agree with human raters at least 80% of the time on a sample of 50 examples before trusting it in CI. It is unreliable when used with vague prompts ('rate this 1 to 10'), uncalibrated against human labels, or run using the same model family as the producer. Use a different model family as your judge, decompose quality into specific dimensions, and always require structured JSON output from the judge.

How many examples do I need in a golden dataset?

50 to 200 human-reviewed examples per major use case. Volume is not the goal. A curated set of 50 examples that covers your real failure modes, edge cases, and representative inputs beats 2000 auto-generated examples. Auto-generated golden sets with unchecked labels produce false confidence, which is worse than having no evals. Grow your dataset incrementally by adding examples from post-incident reviews and new features.

How do evals fit into a CI/CD pipeline for an AI system?

Evals run as a CI step triggered by changes to prompts, model versions, retrieval configs, agent tool definitions, or guardrail logic. The eval runner loads the pinned golden dataset, calls the AI system under test, scores outputs, and compares scores against the baseline. A regression beyond a defined threshold (typically 5 percentage points per category) blocks the merge. Accept new baselines explicitly after intentional improvements. This is identical to the discipline of not merging code that breaks tests.

What tools should I use to run evals in production?

For hosted eval platforms: Braintrust and LangSmith both support golden datasets, LLM-as-judge, and CI integration with good developer experience. Arize Phoenix works well for observability-heavy setups. For self-hosted: build an eval runner in Python that writes scores to a Postgres table, visualize trends in Grafana, and integrate with your existing CI via a GitHub Actions step. The tooling matters less than the discipline. Pick one, use it consistently, and keep your datasets in version control.

Stop Shipping Vibes, Start Shipping Evidence

The difference between AI teams that compound over time and teams that stay stuck in prompt-tweaking loops is evals. Evals are how you build institutional knowledge about your system's behavior. They are how you safely upgrade models, expand capabilities, and onboard new engineers without fear of invisible regressions. They are not optional infrastructure for production AI.

If you are building an LLM-powered system and do not have a systematic eval suite, that is the highest-leverage thing you can fix right now. It is also the work where I spend a significant portion of my time with clients, because getting the eval architecture right early changes how fast a team can move for the next year.

If you want to work through this for your system, whether it is a RAG pipeline, an agent with tool-calling, or a custom LLM integration, you can explore the AI agent development and architecture services I offer or reach out directly at the contact page. I work as an independent architect, not an agency, so the engagement is direct and technical.