Building an Evaluation Harness for AI Agents

An AI agent evaluation harness is the fixed set of real task instances, graders and thresholds you run an agent against on every change, so that a prompt edit, a tool change or a provider model update produces a number rather than an opinion. Without one, every release after the first is a guess.

Key takeaways

• The hard part is the dataset, not the tooling. Thirty to a hundred real task instances with known-good outputs beats any framework applied to invented cases.
• Grade the trajectory as well as the answer. An agent that reaches the right conclusion through the wrong tool calls will reach a wrong one as soon as the inputs shift.
• Use the cheapest grader that works: deterministic checks first, programmatic rules second, a model-as-judge only for what neither can score, and human review for the residue.
• The harness earns its cost the day your provider updates a model you did not pin. That is a change you did not make and cannot see, and the harness is the only thing that detects it.
• A harness that never blocks a release is not a safety net, it is a dashboard. Decide the blocking threshold before you have a result you want to ship.

Most teams discover they need this at the same moment: someone changes a prompt to fix one case, and nobody can say whether anything else broke. Up to that point the agent was evaluated by its builders looking at outputs, which works at ten cases and stops working at forty.

What is an AI agent evaluation harness?

It is a regression test suite for a non-deterministic system, with three parts that have to exist together.

The set — real task instances drawn from the actual workflow, each with the inputs the agent would receive and a recorded known-good result.

The graders — the code and criteria that decide whether a run passed, covering both the final output and the path taken to it.

The runner and its thresholds — the thing that executes the set on demand and in CI, records a score per run, compares it to a baseline, and blocks or permits a release.

Remove any one and the remaining two stop being useful. A set with no runner is a spreadsheet. A runner with no thresholds produces numbers nobody acts on. Graders without a stable set measure different things each time.

Non-determinism is the complication that makes this different from ordinary testing, and it is handled rather than eliminated: run each case a fixed number of times, score the distribution, and treat a case that passes sometimes as a failing case rather than an acceptable one. The variance is itself a finding — a case with high variance is usually a case with an ambiguous task definition.

What goes into the evaluation set, and where do the cases come from?

Real cases from the live workflow, assembled in four passes, each contributing a different kind of coverage.

The third pass is the one that is skipped and the one that matters most. A set built only from a clean recent period reproduces exactly the condition that makes pilots look better than they are — and the data-reality problem described in the PoC-to-production checklist is precisely this gap. Sampling a full historical period, including a close or a quarter-end where the workflow has one, is the cheap fix. The upstream work of getting that data usable at all is covered in what to fix before you build.

The known-good output is the expensive part, and there is no way around it: somebody who knows the work has to say what the right answer was. The economical approach is to have the agent run in shadow mode over the chosen period and have a reviewer confirm or correct its conclusions — the corrections become the set, and the disagreements are the most valuable rows in it. This also means the set is a byproduct of a step you were going to take anyway.

How do you grade an agent’s output when there is no single right answer?

With a ladder of graders, using the cheapest one that can actually decide the case.

Enterprise agent work is more deterministically checkable than the general discussion implies. A reconciliation agent’s output is a match decision with a total; a ticket-triage agent’s output is a category and a priority from a fixed set; a document-extraction agent’s output is a set of typed fields. All of those are gradeable by code, which is faster, free, and does not drift.

Model-as-judge is appropriate for the genuinely open parts — was the explanation adequate, was the escalation reasonable — and carries three obligations if you use it:

• A written rubric with scoring criteria, versioned alongside the prompts.
• A pinned judge model, because an unpinned judge makes your measuring instrument drift under you.
• A periodic audit of the judge against human scores on a sample, with the agreement rate recorded. A judge that has never been audited is an unvalidated instrument.

What do you measure beyond accuracy?

Five dimensions, because an agent that is accurate and unaffordable is not shippable either.

The second row is the one that distinguishes an agent harness from an LLM harness. For a single generation, the output is the thing. For an agent, the output is the end of a path, and two agents with identical accuracy can have completely different reliability depending on whether the path was lucky. Capturing the path is what makes this scoreable, which is an agent observability dependency rather than an evaluation one — if traces do not record tool calls with arguments, trajectory grading is not available to you.

Escalation deserves a note. Teams measure accuracy on completed tasks and ignore the completion rate, which rewards an agent that quietly completes tasks it should have escalated. Score both, and score them on cases where you know which behavior was correct.

How do you run the harness in CI, and what blocks a release?

On every change to anything the agent depends on, with thresholds agreed before the result is known.

Four points on making this operational rather than aspirational.

The scheduled run is not optional. Changes to your agent are not the only changes that affect it. A provider model update, a retrieval index refresh and an upstream API response change all alter behavior with no commit on your side, so the harness runs on a schedule against the live configuration as well as on every change.

Separate the blocking set from the exploratory set. The blocking set is stable, versioned and changes only by deliberate decision. An exploratory set can be anything. Mixing them produces a gate whose threshold moves whenever someone adds a case, which is a gate that means nothing.

Record the full run, not the score. When a release is blocked, the useful artifact is the diff of which cases changed and how — not a number that went from 0.91 to 0.87.

Decide the threshold in advance, in writing. The pressure to ship arrives with the failing run, and a threshold negotiated at that moment is not a threshold.

How do you handle a model version change you did not make?

By pinning where you can, and by detecting where you cannot.

Pin the model version in every call. Where a provider offers a stable alias and a dated version, use the dated version, and treat moving it as a release event with a full harness run and a recorded comparison. This is change control applied to a dependency, and it is the same discipline as pinning a library version.

Where pinning is unavailable — some managed endpoints, some embedded features inside an application platform — you are relying on detection rather than prevention. The scheduled harness run is the detector, and the signal is a score change with no corresponding change on your side. Three things make that signal readable:

• Keep the baseline score and run metadata, including the model identifier the provider reported.
• Run frequently enough that the window between a silent change and its detection is short.
• Alert on the score moving in either direction. A silent improvement is also a change you did not make, and the behavior you validated is no longer the behavior in production.

A related and more common case is retrieval drift: the index changed, the documents are different, and the agent’s grounding moved. The harness catches it the same way, which is an argument for including retrieval-dependent cases in the blocking set. The retrieval-versus-fine-tuning trade-off behind those cases is covered in RAG, fine-tuning and long context for enterprise agents.

How big does the set need to be, and how often is it refreshed?

Large enough that a single case cannot move the score materially, and refreshed on evidence rather than on a calendar.

Thirty to a hundred cases is the practical starting range for a single scoped workflow. Below thirty, one case is several percentage points and the score is noise. Above a hundred, the cost of maintaining known-good outputs starts to dominate unless the grading is fully deterministic — in which case there is little reason not to go further.

Refresh on three triggers rather than quarterly:

1. A production failure that the set did not predict. The case goes in the set the same week. This is the single most valuable row type and it is free.
2. A workflow change. New categories, new entities, a changed chart of accounts or a changed policy all invalidate part of the set.
3. Saturation. When the agent passes everything, the set has stopped measuring the frontier. Add harder cases from the escalation queue rather than declaring success.

What not to do is rebuild the set wholesale, which discards the baseline and makes every historical comparison meaningless. The set is an asset that compounds; treat removals as deliberate and recorded.

The decision

Build the set before the framework. Thirty to a hundred real task instances drawn from a full historical period — representative, known-hard, edge-and-convention, adversarial — with known-good results confirmed by someone who knows the work, is worth more than any tooling choice, and the cheapest way to produce it is a shadow run you were going to do anyway.

Then grade on the lowest rung that can decide the case, score the path and the cost alongside the answer, and write the blocking thresholds down before a release is waiting on them. Run the harness on a schedule as well as on change, because the change most likely to break your agent is one your team did not make — and that run is the only place a silent model update becomes visible.

GrowExx builds and reviews these harnesses through AI agent development and AI code audit.

FAQs