---
title: "Evaluation datasets, graders, tests, and release gates"
chapter: "20"
---

# Evaluation datasets, graders, tests, and release gates

Evaluation is the unit test suite and product measurement system for
probabilistic behavior. Build it before architecture.

## Evaluation layers

1. deterministic: schema, types, citations, permissions, tool parameters;
2. retrieval: recall@k, precision, MRR/nDCG, filter correctness;
3. generation: groundedness, correctness, completeness, style;
4. tools/agents: correct selection, arguments, result use, stop behavior;
5. safety: injection resistance, leakage, harmful or unauthorized actions;
6. operations: latency, tokens, cost, error/timeout rate;
7. product: resolution rate, deflection quality, correction and escalation.

## Dataset

Use production-shaped, privacy-safe examples with easy, typical, hard,
ambiguous, adversarial, multilingual, no-answer, and policy-conflict cases.
Freeze a golden set and maintain a rotating recent set.

## Graders

Prefer executable checks and expert labels. LLM judges scale nuanced review but
have bias and variance. Calibrate against humans, randomize answer order,
separate judge provider/model where useful, and inspect disagreements.

Spring AI provides `RelevancyEvaluator` and `FactCheckingEvaluator`; they are
starting points, not full evaluation platforms.

## Release gate

Compare candidate versus current by slice. Block on critical permission,
citation, tool, or safety failures even if the average improves. Use
confidence intervals for noisy metrics.

## Online monitoring

Sample traces with privacy controls, capture explicit feedback and corrections,
detect distribution drift, and promote failures into the offline suite.

## Feynman check

An AI demo shows one lucky answer. An evaluation shows how often the system is
useful, wrong, unsafe, slow, or expensive across the jobs users actually do.