What is an agent eval?
An agent eval is a repeatable test that scores an agent's behaviour on a fixed set of scenarios, run in CI like any other test suite.
Read the full answer →ANSWERS
6 ANSWERSAgent evals
How to measure whether an agent is working, before and after it reaches production.
How many eval cases do you need for an LLM agent?
Enough that a one-point score move is larger than the noise — usually 200–500 scenarios per agent role, not a handful of spot checks.
Read the full answer →Is an LLM judge good enough, or do you need human evaluation?
An LLM judge is good enough for continuous scoring once it is calibrated against human labels — the humans set the ruler, the model reads it.
Read the full answer →How do you evaluate a multi-agent system?
Score each agent against its own role contract, then score the system end to end — most multi-agent failures are handoff failures, not reasoning failures.
Read the full answer →What metrics should you track for a RAG system?
Split retrieval from generation: recall and precision on the retriever, groundedness and answer quality on the generator. A single end-to-end score hides which half broke.
Read the full answer →When should you start writing evals for an AI feature?
Before the second prompt change. The moment you cannot tell whether an edit helped, you have already needed evals for a while.
Read the full answer →