Building an Evaluation Harness: How to Actually Test if Your AI Agent Works

Introduction

You've built an agent. It can search the web, write code, call APIs, maybe even operate a browser. It works — at least, it worked the three times you tried it yourself. But "it worked when I tried it" is not the same as "it works." Agents fail in ways traditional software doesn't: the same prompt can produce different behavior on different runs, a change that improves one task can silently break five others, and failures often only show up on inputs you didn't think to try.

This is the problem an evaluation harness solves. If an agent harness (covered in the previous article in this series) is the infrastructure that lets a model act in the world, an evaluation harness is the infrastructure that tells you, systematically and repeatably, whether those actions actually work. Without one, you're not testing your agent — you're vibe-checking it.

This article walks through what an evaluation harness actually is, how to build one, and the mistakes that make most homegrown eval setups useless.

Why "It Seems to Work" Isn't Good Enough

Manually testing an agent by chatting with it feels productive, but it breaks down for a few structural reasons:

• Non-determinism: The same prompt can produce different tool calls, different reasoning paths, and different outcomes across runs. A single successful trial tells you very little about the success rate.
• Silent regressions: Changing a system prompt, swapping a model version, or adding a new tool can fix the case you were testing while quietly breaking three others you weren't looking at.
• Narrow coverage: Manual testing naturally drifts toward the inputs you expect to work. The failures that matter — edge cases, ambiguous instructions, adversarial inputs — rarely get tried by accident.
• No comparability: Without consistent tasks and scoring, you can't answer basic questions like "did this change actually help?" or "is this new model better for our use case?"

An evaluation harness turns "does it seem to work" into a number you can track over time, compare across versions, and defend to a team or a stakeholder.

What an Evaluation Harness Actually Consists Of

At a minimum, an eval harness has four parts:

1. A task set — a collection of representative inputs with defined success criteria
2. A runner — infrastructure that executes the agent against each task, safely and repeatably
3. A grader — logic that scores each run, automated, human, or a mix of both
4. A report — aggregated results, logs, and traces that let you see not just the score but why the agent succeeded or failed

Each of these deserves real design attention. Treating any of them as an afterthought is the most common reason eval efforts stall out.

Step 1: Building the Task Set

The task set is the foundation. A harness is only as good as the tasks it runs, and most homegrown evals fail here first.

Start from real usage, not imagination

The best tasks come from actual logs of how people use (or will use) your agent — real questions, real requests, real edge cases. Tasks invented from imagination tend to be too clean; they don't capture the ambiguity, typos, and weird phrasing of real usage.

Cover the failure categories you actually care about

A good task set usually spans several categories:

• Happy path tasks: straightforward requests the agent should reliably handle
• Edge cases: ambiguous instructions, missing information, unusual formatting
• Adversarial/stress tasks: prompt injection attempts, conflicting instructions, requests designed to make the agent take a risky or destructive action
• Long-horizon tasks: multi-step work where errors can compound, testing whether the agent stays on track over many turns
• Regression tasks: specific past failures, added permanently to the set so they never silently reappear

Define success criteria before you run anything

Every task needs a clear definition of what "success" looks like — ideally specific enough that two different people (or two grading passes) would agree on whether a given run passed. Vague criteria like "the response should be helpful" produce inconsistent, low-value scores. Specific criteria — "the returned file must contain a valid CSV with columns X, Y, Z" or "the agent must not delete any file outside the /tmp directory" — produce grading you can actually trust.

Size the set appropriately

You don't need thousands of tasks to start. A well-curated set of 30–100 tasks that actually represents your use cases is far more useful than a bloated set of shallow ones. Grow the set over time, primarily by adding real failures you encounter in production.

Step 2: The Runner

The runner is the infrastructure that actually executes the agent against each task. Key design considerations:

Isolation between runs

Each task should run in a clean, isolated environment — a fresh sandbox, container, or virtual filesystem — so that one task's side effects (files created, state changed) can't leak into another. Without this, results become unreliable and hard to reproduce.

Repeatability

Because agents are non-deterministic, running each task once and calling it done is misleading. Serious evaluation setups run each task multiple times (often 3–10 repetitions) and report a success rate, not a single pass/fail. A task that passes 9 times out of 10 is meaningfully different from one that passes 10 out of 10, even though a single run wouldn't tell you the difference.

Realistic tool access

The runner should give the agent access to the same (or equivalent, sandboxed) tools it would have in production — search, code execution, file access — rather than a stripped-down test-only version. Evaluations against a fake or simplified toolset tend to overstate how well the agent will perform for real.

Timeout and budget enforcement

Tasks should have defined limits on steps, time, and cost, matching what you'd enforce in production. An agent that "succeeds" only by taking 200 tool calls and $40 of API spend hasn't really succeeded for most practical purposes.

Step 3: Grading

Grading is where most eval efforts either become genuinely useful or quietly become theater. There are three broad approaches, and most serious harnesses use a mix of all three.

Rule-based / programmatic grading

For tasks with objectively checkable outcomes — did the code pass the test suite, does the output file have the right schema, did the agent avoid calling a forbidden tool — automated checks are fast, cheap, consistent, and should be your default whenever a task allows it.

Model-based grading ("LLM-as-judge")

For more open-ended tasks — is this summary accurate, is this email appropriately toned, did the agent's reasoning make sense — a separate model call can grade the output against a rubric. This scales far better than human grading, but it comes with real caveats:

• The grading model needs a specific, detailed rubric, not a vague instruction — the same specificity problem as writing task success criteria in the first place
• Model-based grading has its own error rate and biases (e.g., favoring longer or more confident-sounding answers), so it should be periodically spot-checked against human judgment
• It works best combined with rule-based checks where possible, rather than as the only signal

Human grading

For high-stakes or genuinely ambiguous tasks, human review remains the gold standard — and the source of truth you use to validate whether your automated graders are actually trustworthy. It doesn't scale to every run, but a regular sample of human-graded results is what keeps automated grading honest over time.

Step 4: Reporting and Traceability

A score alone isn't very actionable. The most valuable part of a mature eval harness is often the trace — a full record of every reasoning step, tool call, and observation the agent made during a task. When a task fails, the trace is what tells you why: did the agent misunderstand the instructions, pick the wrong tool, get tripped up by a bad tool response, or lose track of the goal over a long context?

Good reporting typically includes:

• Aggregate pass rate by category (happy path, edge case, adversarial, etc.)
• Trends over time, so you can see whether changes are actually helping
• Per-task drill-down with full traces for any failure
• Cost and latency metrics alongside correctness — a harness that only measures "did it work" and ignores "how expensive was it" will miss real regressions

Common Mistakes in Building Eval Harnesses

Testing only the happy path. If every task in your set is a clean, well-formed request, your eval will tell you the agent works great right up until it meets a real user.

Treating a single run as the answer. Given non-determinism, one pass or one fail tells you almost nothing. Always run multiple trials per task before drawing conclusions.

Vague grading criteria. If two people would disagree on whether a given output passed, your scores aren't measuring what you think they're measuring.

Never updating the task set. Production will surface failure modes you didn't anticipate. Every real failure is a candidate for a permanent regression test — skipping this means you'll keep re-discovering the same bugs.

Ignoring cost and latency. An agent that "passes" every task but takes ten minutes and costs several dollars per run may not actually be viable in production. Track these alongside correctness from day one.

Building the harness once and never running it. An eval harness only creates value if it's actually run — ideally automatically, on every meaningful change to the prompt, tools, or model — not just when something has already gone wrong.

A Minimal Example Structure

For teams starting from scratch, a lightweight but real eval harness might look like:

1. A folder of task definitions (input, expected behavior, grading logic) as simple structured files
2. A script that spins up a sandboxed environment per task, runs the agent, and captures the full trace
3. A grading step that runs programmatic checks where possible and falls back to an LLM-as-judge call with a detailed rubric for the rest
4. A simple report — even a spreadsheet or dashboard — showing pass rate by category over time, with links to failing traces

This doesn't require a specialized framework to start. The discipline of defining tasks clearly, running them repeatedly, and tracking results over time matters more than the specific tooling.

Conclusion

Building an agent that works once is easy. Building one you can trust — and prove you can trust — requires an evaluation harness that treats testing with the same rigor as the agent itself. The teams that ship reliable agents aren't the ones with the cleverest prompts; they're the ones who built a feedback loop that tells them, honestly and repeatedly, when something breaks. Eval harnesses aren't the exciting part of agent development, but they're the part that determines whether what you build actually holds up once real users — and real edge cases — get their hands on it.