Press ESC to exit fullscreen
📖 Lesson ⏱️ 60 minutes

Why Agent Evaluation Is Different

Why model metrics don't transfer to agents — non-determinism, multi-step trajectories, side effects, and the compounding-error math that makes a 95%-per-step agent fail most tasks

The Vibe Check Ceiling

Here’s how most agents get evaluated in their first month. An engineer writes ten queries they think are representative, runs the agent, reads the outputs, and decides it “looks good.” When they change the prompt, they rerun some of those queries, read again, and decide it “looks better.”

This works surprisingly well at first. It stops working at a predictable point. Call it the vibe check ceiling, and you hit it when one of these becomes true:

  • More than one person changes the agent, and they disagree about whether a change helped
  • The agent handles more distinct task types than one person can hold in their head
  • A model upgrade ships and nobody can say whether it’s safe to adopt
  • A customer reports a failure, and you can’t tell if it’s new or has always been there

Past this ceiling, every change is a coin flip you can’t see the result of. An evaluation framework is how you get past it. Before you build one, you need to understand why the tools you already know from ML evaluation aren’t enough.

Model Evaluation, Briefly

Classic model evaluation has a clean shape:

  1. A fixed input
  2. A single output
  3. A label to compare against
  4. A metric (accuracy, F1, BLEU, ROUGE) that aggregates over a test set

Even LLM benchmarks mostly keep this shape: a question, an answer, a reference, a score. The model is a function. You evaluate the function.

An agent isn’t a function. It’s a loop. It reads a goal, decides on an action, observes the result, and decides again, often a dozen times, while changing the state of the world as it goes. Every property that made model evaluation tractable breaks somewhere in that loop.

Five Properties That Make Agents Hard to Evaluate

1. Non-Determinism at Every Step

Run the same model call twice at temperature 0 and you’ll often get the same answer. Run the same agent twice and small differences in one step change which tool gets called next, which changes the observation, which changes everything after. A single run tells you what can happen, not what usually happens.

This means one run per test case isn’t a measurement. Lesson 10 covers how many runs you need and how to report the variance. For now, accept that agent evals are inherently statistical.

2. Many Valid Paths

Ask Quill (our billing-support agent) “Why was I charged twice in March?” One good trajectory:

lookup_customer → get_invoices → (finds duplicate) → issue_refund → respond

Another equally good one:

lookup_customer → get_invoices → search_kb("duplicate charge policy") → issue_refund → respond

A third that’s arguably better for a high-value account:

lookup_customer → get_invoices → escalate_to_human → respond

There’s no single “label” trajectory. Exact-match comparison against one reference path punishes correct behavior. Lesson 7 shows how to grade trajectories against constraints (must call X before Y, must never call Z) instead of exact sequences.

3. Side Effects

When a classifier is wrong, you get a wrong label. When Quill is wrong, a customer gets $180 they weren’t owed, or a legitimate refund doesn’t happen. Agents act, and the actions are often the thing you most need to evaluate.

Two consequences follow:

  • You need to evaluate state changes, not just text. Did the refund record get created, for the right invoice, for the right amount?
  • You need sandboxed or mocked environments so evaluation doesn’t refund real money. Lesson 2 covers how to architect for this.

4. Errors Compound

This is the property that surprises people most, so it gets its own section below.

5. The Final Answer Hides the Process

An agent can produce the right final answer by luck: it guessed the invoice instead of looking it up, and the guess happened to be right. It can also produce a reasonable-sounding wrong answer after a perfectly good process, because one tool returned stale data. If you only grade the final answer, you can’t tell these apart, and you can’t fix what you can’t see.

The Compounding-Error Math

Suppose each step of an agent’s trajectory succeeds independently with probability p. A task that needs n steps succeeds with probability p to the power n.

for p in [0.99, 0.95, 0.90]:
    for n in [5, 10, 20]:
        print(f"per-step {p:.0%}, {n:>2} steps → task success {p**n:.1%}")
per-step 99%,  5 steps → task success 95.1%
per-step 99%, 10 steps → task success 90.4%
per-step 99%, 20 steps → task success 81.8%
per-step 95%,  5 steps → task success 77.4%
per-step 95%, 10 steps → task success 59.9%
per-step 95%, 20 steps → task success 35.8%
per-step 90%,  5 steps → task success 59.0%
per-step 90%, 10 steps → task success 34.9%
per-step 90%, 20 steps → task success 12.2%

An agent that’s “95% accurate” at each decision fails four out of ten ten-step tasks. Real agents aren’t quite this bad, because they can notice errors and recover, which breaks the independence assumption. But that recovery is itself a behavior you have to measure.

The practical lesson has two parts:

  1. Step-level metrics are necessary but not sufficient. A 2% improvement in tool-selection accuracy can be a 15% improvement in task success on long tasks. You need both numbers.
  2. Long tasks need disproportionate eval attention. They’re where compounding bites, and they’re usually underrepresented in hand-written test sets because they’re tedious to write.

Three Things You Can Evaluate

Every agent eval measures some mix of three things. Being explicit about which one you’re measuring avoids most confused eval designs.

DimensionQuestionExample for QuillTypical grader
OutcomeDid the world end up in the right state?A refund for exactly $49 exists on invoice INV-2291State assertion
TrajectoryDid the agent take an acceptable path?Looked up the customer before refunding; never refunded twiceTrajectory constraints
ResponseWas the communication good?Clear, accurate, polite, no invented policyRubric / LLM judge

Outcome is usually the most important and the easiest to grade reliably, if your architecture exposes the state. Trajectory catches lucky successes and wasteful paths. Response quality is where LLM judges earn their keep, and also where they’re most dangerous if uncalibrated.

Anti-Patterns Before a Framework Exists

You’ll recognize most of these. Each one is a symptom the rest of the course fixes.

The demo set. Ten hand-picked queries that the agent was tuned on. They measure memorization of the demo, not capability. Fixed in Lesson 4.

Final-answer string matching. assert "refund" in response. Passes when the agent says “I can’t issue a refund.” Fixed in Lesson 5.

The unvalidated judge. “We use GPT-4 as a judge, it gives us 8.2/10.” Nobody has checked whether the judge agrees with a human. Fixed in Lesson 6.

Single-run scoring. A change goes from 82% to 85% on one run of 40 cases. That’s well inside the noise. Fixed in Lesson 10.

Eval in a notebook. The eval exists, but someone has to remember to run it, so it runs before big launches and never in between. Fixed in Lesson 12.

Offline-only. The test set was written once, in January. Production traffic in November looks nothing like it. Fixed in Lesson 11.

What a Mature Eval Practice Looks Like

By the end of this course, the target looks like this:

  • A versioned golden dataset of a few hundred cases, stratified by intent and risk, that grows every week from production failures
  • A grader stack where cheap deterministic checks carry most of the weight, and a calibrated judge handles what’s left
  • Every run traced, so a failing case can be opened and inspected step by step
  • Reliability reported honestly, with multiple trials, confidence intervals, and cost and latency alongside quality
  • A CI gate that runs a fast suite on every pull request and a full suite nightly
  • Online evaluators watching sampled production traffic, feeding new cases back into the dataset

None of that is exotic. It’s ordinary engineering discipline applied to a system that happens to be non-deterministic. The hardest part is usually the first step: designing the agent so it can be observed at all. That’s the next lesson.

Key Takeaways

  • An agent is a loop with side effects, not a function, so model-evaluation habits don’t transfer
  • Non-determinism means a single run isn’t a measurement; agent evals are statistical
  • Many trajectories can be valid, so grade against constraints, not one reference path
  • Per-step errors compound: 95% per step is about 60% task success over ten steps
  • Separate outcome, trajectory, and response quality, and pick graders for each deliberately

Exercise

Take an agent you’ve built (or use Quill’s tool list from the course overview). Write down:

  1. The three most common task types it handles, and the typical number of steps for each
  2. For each, what the correct outcome is in terms of state, not text
  3. One trajectory that reaches the right outcome but that you’d still consider a failure, and why

Keep this list. You’ll turn it into the seed of a golden dataset in Lesson 4.