Agent evaluation (Evals)

Agent evaluation, usually shortened to evals, is the practice of measuring an AI agent against a fixed set of test cases, each with a defined notion of what a correct result is. It replaces "it looked good in the demo" with a number that can be compared across versions, across models and across weeks in production.

Why is testing an agent different from testing software?

Ordinary software is tested with one input, one expected output, one assertion. An agent given the same input twice can take different routes and produce different wording. So the test asks a different question. Instead of "is the output equal to X" it asks "does the output satisfy the criteria": the right category, the correct total, no invented field, a handoff to a person where one was required.

Because a single case can pass by luck, evals run many cases and report a rate. One good example proves nothing. Eighty cases with a completion rate, a handoff rate and a list of the failures is a result you can act on.

What does an eval suite consist of?

Cases. Real inputs from the process, anonymised where needed, plus deliberately awkward ones: missing data, two documents that contradict each other, an email that tries to give the agent instructions of its own (see prompt injection).

Expected results. For deterministic parts an exact value: the CRM record exists, the total matches. For text a rubric: complete, correct tone, no claims that are not in the source.

Graders. Code wherever the check can be written as code. For language quality a second model acts as judge, and that judge is itself calibrated against human ratings on a sample before anyone trusts its scores.

Reporting. Per version of the agent, per model, and per node if the agent is built as an agent graph, so a regression can be located rather than merely noticed.

Why does this matter for an organisation deploying agents?

Evals matter at three moments.

Before rollout. The go or no-go for an AI pilot becomes a number from the suite, measured against the same cases a person handled last quarter, rather than an impression from a demo.

At model changes. Providers retire models and replace them on their own schedule. With a suite you know in an afternoon whether the replacement is safe for your process. Without one, your customers tell you over the following weeks.

In operation. A sample of production cases runs through the graders every week. A falling score is the earliest signal that something changed upstream: a new document layout, a renamed field in the CRM, a supplier that switched formats.

This is also where accountability becomes possible. An agent whose outcome is measured can have an owner, because the number sits in a report and a maintenance agreement can name it. An agent nobody measures cannot be owned by anyone.

Example: evaluating a quote-preparation agent

The agent reads an incoming request, looks up prices and drafts a quote. The suite holds 120 historical requests together with the quote a person actually sent. Code graders check that the line items match and that the total is within tolerance. A model judge, calibrated on 30 human-rated samples, scores completeness and tone.

Fifteen further cases are adversarial. A request for a product that is not in the price list must end in a handoff, not in an invented item. A request with contradictory quantities must produce a clarifying question rather than a guess.

The report shows completion rate, handoff rate with reasons, and cost per case, per agent version. When a new model is released, the suite runs against it before a single customer sees the result.

What evals cannot tell you

Evals cover the cases you thought of. The cases you did not think of arrive in production as handoffs for a new reason, and every one of them belongs in the suite. A suite that has not grown in three months is a suite that has stopped learning from the process it measures.

Nor do evals replace human-in-the-loop. They tell you how often the agent needs a person. They do not remove the need. How this fits into the wider engineering discipline is covered in Shipping AI that survives production. In a managed agent, the suite is part of the deliverable, not a separate project; see agents.