AI agent evals are the tests that tell you whether an agent is doing its job well. They exist because the way software teams normally test things, asserting that a given input produces a known output, falls apart on a system that never produces the same output twice. A unit test checks for one correct answer. An agent has no single correct answer to check against, so the real question becomes how you judge quality with no fixed target to aim at. Most teams reach production before they have answered it.
You see the gap the first time someone writes a test for an agent the way they would for a function. They capture a known-good output, assert that the agent reproduces it, and the test goes green. Then it fails on the next run. The agent has not got worse; it has phrased the same correct answer a different way. Pinning behaviour to a fixed string is the right instinct for deterministic code and the wrong one here.
The Test That No Longer Applies
Deterministic software has a clean definition of correct. Give it an input and there is a specific output it should produce, and a test asserts that it does. The assertion is binary and it is stable. It passes today, it passes tomorrow, and the day it fails it tells you something broke.
A language-model agent gives you none of that. The same input produces different outputs from one run to the next, shaped by sampling, by the context it was handed, and by whatever version of the model sits behind the API that day. There is usually no single correct response. There is a range of acceptable ones and a range of unacceptable ones, and the line between them is a matter of judgement, not string equality. You cannot assert your way to a line like that. The unit test here has not got harder to write; it has stopped being the right tool.
You cannot assert a correct answer when there is no single correct answer. All you can do is judge the output against a standard you set in advance. That is a different kind of test, and most teams have never built one.
What an Eval Actually Is
An eval judges the quality of an output against a standard you have defined. The question moves from "did it return exactly this?" to "was this output good enough on the things we care about?" That second question has a precondition most teams have never met: someone has to say what good means here. Faithful to the source? Within policy? Correctly formatted? You cannot write the eval until those are named, and writing the eval is often the moment a team discovers it never agreed on them.
An eval does not hand you a single pass or fail. It gives you a measurement you can track, a baseline to hold the system to as the model, the prompt and the code around it all shift underneath you. Without that baseline, "the agent got better" is a claim you are asking people to take on faith. The baseline turns it into a number you can point at.
The Three Ways to Judge
In practice, judging agent output draws on three methods, layered by what each one can see and what it costs.
Deterministic checks handle the structured surface of an output. Did it parse as valid JSON? Did it call a tool it was allowed to call? Did it stay inside the schema and the policy? These checks are cheap, fast and reliable, and they only reach the part of behaviour you can write down as a rule. They confirm the output was well-formed. They say nothing about whether it was right.
Model-graded evals use a language model to score the parts no rule can reach: whether a summary stayed faithful to its source, whether a response answered the question, whether the tone fit. The appeal is scale, since a model can judge volumes of output no human could read. The catch is that the judge is itself a probabilistic system, and it can be inconsistent or quietly gamed. Treating its score as ground truth is one of the most common mistakes teams make. Why it goes wrong is the subject of the LLM-as-judge problem.
Human annotation is the anchor. A small set of examples, graded carefully by a person, calibrates everything else: it tells you whether your deterministic checks catch the cases that matter, and whether your model judge agrees with a human often enough to be trusted. It does not scale, and it is not meant to. Its job is to be the ground truth the cheaper methods are measured against.
The Path Matters More Than the Answer
This is where judging an agent parts company with judging a single model call. A one-shot LLM eval looks at an input and an output and scores the output. An agent does not produce one output. It works in steps: it calls a tool, reads the result, decides what to do next, and repeats. The answer you see is only the last step. Score that alone and you miss everything that produced it.
An agent can reach the right answer through a broken run. It loops through the same lookup a dozen times. It calls a tool it should never have been able to reach. It breaks a policy halfway through, in a way that happens not to change the final text. Grade only the answer and every one of those passes. The fault is in the path, and to grade the path you have to be able to see it, which means you need goal-level observability first. You cannot judge a run you cannot reconstruct.
Where Eval Programmes Go Wrong
The ways these programmes fail are consistent enough to name. Most teams grade only the final output, which keeps unsafe and wasteful runs invisible for as long as the answer comes out right. Many collapse everything into a single quality score, one number that folds faithfulness, safety and format together so that when it drops you cannot tell which of them slipped. Some trust a model judge with nothing behind it to confirm its scores track reality. And almost everyone builds their evals late, after an incident has already shown them what they should have been measuring.
None of these is a tooling problem. You cannot buy your way out with an eval platform, because the platform can run the evals but it cannot tell you what to evaluate. That decision, what good looks like and how you hold the system to it, is yours.
A Practical Starting Point
The first useful move involves no instrumentation at all. Take one real task your agent performs and write down what it would take to judge a good run of it: the things that have to be right, the failures you most need to catch, and a handful of real cases, including the ones that have already gone wrong. That description is the specification for your eval set. It beats any amount of generic metric collection, because it is built from the way your system actually fails, not from whatever happens to be easy to measure.
From there it is engineering. You anchor a human-graded golden set, run deterministic checks underneath model-graded scoring, and wire the whole thing into a baseline you can run on every change. Building that rig, the thing that holds a non-deterministic system to a standard, is the work we take apart in our AI Agent Observability and Reliability course. This piece argues for doing it; the course shows how. The teams whose agents stay reliable in production are the ones that decided what good meant and measured against it from the start. The rest find out what they should have measured the first time an agent is confidently, expensively wrong.
Frequently Asked Questions
Why can't you unit-test an AI agent?
A unit test asserts that a given input produces one specific output, which works because deterministic code returns the same result every time. A language-model agent does not: the same input gives different outputs from run to run, shaped by sampling, context and the model version behind the API. There is rarely one correct response, only a range of acceptable ones and a range of unacceptable ones, so you cannot pin the behaviour to a fixed string. You judge an agent against a standard instead of asserting it against an answer. That is what an eval does.
What is the difference between LLM evals and AI agent evals?
An LLM eval looks at one input and one output and judges the output. An agent eval has to judge a sequence of steps, because an agent calls tools, reads results and decides what to do next; the answer you see is only the last step. This matters because an agent can reach a correct answer through a broken run, looping wastefully, calling a tool it should not have, or breaking a policy partway through, and grading only the final output hides all of it. Agent evals judge the whole trajectory, which means you first need to be able to reconstruct the run.
What is a model-graded eval, or LLM-as-a-judge?
A model-graded eval uses a language model to score output that no fixed rule can check, such as whether a summary stayed faithful to its source or whether a response actually answered the question. Its value is scale: it can judge volumes no human could review. Its risk is that the judge is itself a probabilistic system that can be inconsistent or gamed. Treating a model-graded score as ground truth, with no human-graded set to calibrate it, is one of the most common evaluation mistakes.
How do you start evaluating AI agents?
Take one real task your agent performs and write down what it would take to judge a good run of it: the quality dimensions that matter, the failures you most need to catch, and a handful of real cases, including ones that have already gone wrong. That description is the specification for your eval set, and it is grounded in how your system actually fails rather than in what is easy to measure. From there you anchor a small human-graded set, run deterministic checks under model-graded scoring, and keep the result as a baseline you run on every change.