Why AI Evaluation Is Different From Traditional Testing

Software testing assumes deterministic outputs. AI evaluation does not. Two correct answers to the same prompt can look completely different, and two wrong answers can look identical to a correct one. The evaluation problem is fundamentally a measurement problem: how do you score a probabilistic system against a target distribution of behavior, repeatably, across model and prompt changes?

• Outputs are non-deterministic: same input can produce different valid outputs across runs
• Quality is multidimensional: correctness, helpfulness, safety, format, latency, cost - all must be scored
• Ground truth is fuzzy: many tasks have no single right answer, only better and worse responses
• Evaluation must scale: a 5,000-call regression cannot rely on human review
• Evaluators are themselves AI: LLM-as-judge introduces its own bias and calibration problem
• The stakes are operational: a 2% regression on a slice that handles your highest-value users is a production incident, not a metric blip

The Golden Dataset: Where Evaluation Actually Starts

No serious AI evaluation exists without a frozen golden dataset. It is the unit test suite of the LLM era. Teams that try to evaluate against a moving target produce numbers nobody trusts. Teams that build a 200-1000 example golden set from real production traffic and freeze it can compare every change like-for-like, forever.

• Draw examples from real production traffic, not synthetic ideals or vendor demos
• Cover the distribution: common cases, long tail, edge cases, adversarial inputs, refusal cases
• Slice the dataset: tag by use case, user tier, language, complexity, sensitivity - score per slice not just aggregate
• Size: 50-200 to start, grow to 500-2000 once the structure is right. Quality over volume
• Freeze it: changes to the golden set are themselves versioned and reviewed, not casual edits
• Label format: expected output, expected refusal, rubric criteria, or all three depending on task type
• Refresh cadence: add 20-50 examples per quarter as production distribution shifts, but never remove without justification
• Provenance: every example needs a source link or trace ID so you can debug regressions back to the original

Rubric Design and Scoring Methods

A rubric is the operational definition of "good." Bad rubrics are the most common reason eval pipelines produce numbers nobody believes. The best rubrics are concrete, binary or low-cardinality (3-5 point scales), and tied to user-visible behavior, not internal aesthetics.

• Deterministic checks first: schema validity, regex matches, length bounds, required field presence - cheap, fast, no model needed
• Reference-based: exact match, substring match, embedding similarity, BLEU, ROUGE for tasks with reference outputs
• Reference-free rubrics: helpfulness, correctness, safety, tone, format - scored by LLM-as-judge or humans
• Binary rubrics beat fuzzy scales: "did this answer the question yes/no" beats "rate helpfulness 1-10"
• Decompose multi-criteria into independent rubrics: correctness AND format AND safety, scored separately
• Calibration: every rubric scored by an LLM judge needs a human-labeled calibration set, with agreement measured
• Inter-judge agreement: when judges (human or model) disagree more than 15-20%, the rubric is broken
• Avoid composite scores in the early days - per-dimension scores are easier to debug than a single number

LLM-as-Judge: The Workhorse, Used Correctly

LLM-as-judge is the only way to scale evaluation to thousands of examples per CI run. Done well, it produces scores that correlate strongly with human labels. Done badly, it produces confident-looking nonsense that hides real regressions. The difference is calibration and rigor, not the choice of judge model.

• Use a strong judge model (Claude Opus, GPT-5 class, or Gemini Pro). Cheap judges produce noisy scores
• Calibrate against humans: hand-label 50-200 examples, measure judge agreement, iterate the rubric until agreement >85%
• Pairwise beats pointwise: ask the judge "A or B" rather than "rate A 1-5." Higher signal, lower variance
• Position bias: judges favor the first answer presented. Always randomize order or run both orderings
• Self-preference bias: a judge prefers outputs from its own family. Use a different family for the judge when possible
• Chain-of-thought judges: ask the judge to reason before scoring. Improves correlation with human labels
• Multi-judge ensembles: 2-3 judges with majority vote when stakes are high
• Audit the judge: spot-check 5-10% of judge scores monthly against humans, retrain the rubric on disagreements

Regression Gates and CI Integration

Evaluation that is not wired into the deployment pipeline is theater. The point of evals is to block bad changes before they ship. Production teams have a hard ship-gate: a new prompt, model, or fine-tune cannot be promoted unless it beats the current baseline on every guarded slice of the golden set.

• CI runs the eval suite on every PR that touches prompts, models, or AI logic
• Baseline scoring: the current production version is re-scored on the same dataset for every comparison
• Per-slice gates: candidate must beat or tie baseline on every protected slice, not just on average
• Cost and latency gates: candidate must stay within budget bands, regressions block deploy
• Confidence intervals: small datasets and noisy judges produce noisy scores - require statistically meaningful delta before promotion
• A/B canaries in production: even after offline evals pass, run new version on 1-5% of live traffic for 24-72 hours
• Rollback discipline: previous prompt/model/version must be one config flip away
• Eval flakiness budget: if the same eval re-runs vary >5%, fix the eval before trusting its verdict on changes

Live Sampling and Production Observability

A golden set catches known failure modes. Live sampling catches the ones you have not seen yet. Every serious production AI system samples a percentage of live traffic, scores it against the same rubrics, and surfaces anomalies for human review. This is the only way to catch the failures the golden set was not designed for.

• Sample 1-5% of live production traffic, score against the same rubrics as the golden set
• Stratified sampling: oversample high-value, high-risk, or low-frequency slices
• Trace logging: capture full request/response/tool-call traces, not just final outputs. Required for debugging
• Privacy and PII: log with redaction, retention windows, access controls. Required for compliance and trust
• Tools that do this well: Braintrust, LangSmith, Langfuse, Helicone, Arize, Logfire, Honeycomb with LLM extensions
• Anomaly alerts: drop in eval scores, spike in refusals, latency regression, cost spike. Page on real signals not vanity metrics
• Feedback loops: production errors flow back into the golden set as new test cases. The eval suite grows with the product
• User feedback: thumbs up/down, written complaints, and support tickets are evaluation data - capture and label them

Drift Detection and Continuous Evaluation

AI systems drift even when nothing changes on your side. Model vendors silently update weights. Retrieval indexes go stale. User behavior shifts. The golden set scored monthly tells you when the system is degrading even if no PR has landed. The earlier you catch drift, the cheaper the fix.

• Re-score the production baseline on the golden set weekly or monthly, even without changes
• Track per-slice scores over time, alarm on statistically significant drops
• Distribution drift: monitor input embedding centroids and topic distributions over time
• Vendor weight changes: pin model versions explicitly (gpt-5.4-2026-04-12 not gpt-5.4-latest) and watch behavior on version bumps
• Tool drift: every tool the agent calls has its own contract. Schema changes upstream break agents silently
• Refusal rate drift: a 2-point monthly rise in refusal rate often signals a vendor safety update degrading your use case
• Cost drift: token usage per request creeps up as prompts grow. Track and trim
• Quarterly full re-evaluation: re-baseline the golden set scores against current production, document deltas

Common Mistakes That Invalidate Evaluation

Most failed eval setups fail in the same handful of ways. Catching these early saves months.

• No frozen golden set: every comparison is against a moving target, numbers mean nothing
• Synthetic eval data only: scores look great until real users hit edge cases the synthetic set never covered
• Uncalibrated LLM judge: confident scores with no human-correlation check. Often inverts on critical slices
• Aggregate scores only: a 90% average can hide a 30% regression on the slice that matters most
• Eval set training contamination: examples leak into prompts, few-shot, or fine-tuning data. Inflates scores fraudulently
• No cost/latency tracking in evals: ship a quality win that doubles cost or breaks SLA
• No CI integration: evals run quarterly by hand. Regressions reach production
• No live sampling: golden set catches known failures only, blind to unseen ones
• Single-author rubrics: rubric reflects one person's taste, no team agreement, drifts every quarter

Working With Me on an Evaluation Engagement

Most teams who bring me in for evaluation design are post-launch, dealing with reliability complaints, and lacking a defensible answer to "is this getting better or worse." The engagement converts that into a measurable system: a golden set, rubrics, judge calibration, CI gates, live sampling, and drift monitoring. It is the foundation everything else (prompt iteration, fine-tuning, model selection, agent reliability) rests on.

• Week 1: audit current AI system, instrument logging, identify failure modes from real complaints
• Week 2-3: build the golden dataset from production traffic, design rubrics, calibrate the LLM judge against humans
• Week 4-5: CI integration, regression gates, baseline scoring of current production
• Week 6: live sampling pipeline, dashboards, alerting on score and cost drift
• Week 7-8: drift monitoring playbook, quarterly review cadence, team training and handoff
• Deliverables: golden dataset, rubric library, calibrated judge prompts, CI configs, observability dashboards, runbook
• Typical engagement: 6-10 weeks for a focused product, longer for multi-feature platforms

FAQ