What is an agent eval?
An agent eval is a repeatable test that scores an agent's behaviour against a fixed set of scenarios. Unlike a unit test it grades a distribution of outputs rather than one exact string, so it reports a score and a confidence interval instead of pass or fail. Evals are the only practical way to know whether a prompt, model, or tool change made an agentic system better or worse.
A usable eval suite has three layers. Trajectory checks assert that the agent called the right tools in a defensible order. Outcome checks grade the final answer against a rubric, usually with a stronger model as judge plus a human-labelled calibration set. Regression budgets fail the build when a score drops more than an agreed margin below the last release.
The part teams skip is the calibration set. An LLM judge that has never been checked against human labels will drift with the judge model itself, which reintroduces exactly the problem the eval was supposed to catch. Budget a few hundred hand-labelled examples per agent and re-verify the judge against them whenever you change judge models.
Scenarios should come from production traffic, not from imagination. Sampling real sessions, clustering them by intent, and promoting the hard clusters into the suite is what makes the score track user-visible quality.
Written by Binary AI Labs · Reviewed