Evaluation datasets, graders, tests, and release gates

Evaluation is the unit test suite and product measurement system for probabilistic behavior. Build it before architecture.

Evaluation layers

1. deterministic: schema, types, citations, permissions, tool parameters;
2. retrieval: recall@k, precision, MRR/nDCG, filter correctness;
3. generation: groundedness, correctness, completeness, style;
4. tools/agents: correct selection, arguments, result use, stop behavior;
5. safety: injection resistance, leakage, harmful or unauthorized actions;
6. operations: latency, tokens, cost, error/timeout rate;
7. product: resolution rate, deflection quality, correction and escalation.

Dataset

Use production-shaped, privacy-safe examples with easy, typical, hard, ambiguous, adversarial, multilingual, no-answer, and policy-conflict cases. Freeze a golden set and maintain a rotating recent set.

Graders

Prefer executable checks and expert labels. LLM judges scale nuanced review but have bias and variance. Calibrate against humans, randomize answer order, separate judge provider/model where useful, and inspect disagreements.

Spring AI provides RelevancyEvaluator and FactCheckingEvaluator; they are starting points, not full evaluation platforms.

Release gate

Compare candidate versus current by slice. Block on critical permission, citation, tool, or safety failures even if the average improves. Use confidence intervals for noisy metrics.

Online monitoring

Sample traces with privacy controls, capture explicit feedback and corrections, detect distribution drift, and promote failures into the offline suite.

Feynman check

An AI demo shows one lucky answer. An evaluation shows how often the system is useful, wrong, unsafe, slow, or expensive across the jobs users actually do.