Course Content
Designing the Evaluation Framework
The four evaluation layers — component, step, trajectory, and outcome — plus the core abstractions every framework needs: Case, Run, Grader, Score, Report
Why a Framework, Not a Script
Your first eval is usually a script: loop over some inputs, run the agent, check something, print a percentage. That’s the right way to start. It stops scaling as soon as you want to:
- Add a second kind of check without rewriting the loop
- Compare two agent versions on the same cases
- Run each case several times and report variance
- Find out which check failed on which case, and open its trace
An evaluation framework is the small set of abstractions that make all of that routine. It doesn’t need to be big. The version we build in this lesson is about 200 lines, and it’s the backbone of everything that follows.
The Four Evaluation Layers
Agents fail at different granularities. A good framework evaluates at all four, because each layer catches failures the others miss.
┌────────────────────────────────────────────────────────────┐
│ 4. OUTCOME Did the task succeed end to end? │ few, expensive, decisive
├────────────────────────────────────────────────────────────┤
│ 3. TRAJECTORY Was the path acceptable, safe, efficient? │
├────────────────────────────────────────────────────────────┤
│ 2. STEP Was each individual decision correct? │
├────────────────────────────────────────────────────────────┤
│ 1. COMPONENT Does each piece work in isolation? │ many, cheap, diagnostic
└────────────────────────────────────────────────────────────┘Layer 1: Component
Test pieces in isolation: a retrieval function returns the right KB article; a tool’s argument parser handles malformed dates; the intent classifier routes “I was charged twice” to billing. These are close to ordinary unit tests. They’re fast and they localize failures precisely.
Layer 2: Step
Freeze the conversation at a specific point and ask: given this exact context, does the model make the right next decision? For Quill: given a message history where get_invoices just returned two identical charges, does the next call refund the duplicate rather than the original?
Step evals are the agent equivalent of unit tests for decisions. They’re cheap (one model call each), deterministic in their setup, and the best tool for regression-testing a specific bug fix.
step_case = {
"messages": [...], # history up to the decision point
"expect_tool": "issue_refund",
"expect_args": {"invoice_id": "INV-2291"},
}Layer 3: Trajectory
Run the full loop and grade the path: required tools called, forbidden tools avoided, correct ordering, no redundant calls, within a step budget. Covered in depth in Lesson 7.
Layer 4: Outcome
Run the full loop and grade the end state and response. Did the right refund happen? Was the customer told accurately? This is the number stakeholders care about.
The common mistake is having only Layer 4. Your outcome score drops from 88% to 81%, and you have no idea why. With step and trajectory layers, you can see that tool-selection accuracy after get_invoices fell, which points directly at the prompt change that caused it.
The Five Core Abstractions
Case ──▶ Runner ──▶ Run ──▶ Grader(s) ──▶ Score(s) ──▶ ReportCase
One test scenario: input, setup state, expectations, and metadata for slicing results.
from dataclasses import dataclass, field
from typing import Any
@dataclass(frozen=True)
class Case:
id: str
input: str
initial_state: dict
expected: dict = field(default_factory=dict) # free-form, read by graders
tags: tuple[str, ...] = () # e.g. ("intent:duplicate", "risk:high")
split: str = "regression" # "smoke" | "regression" | "adversarial"expected is deliberately loose. Different graders read different keys: expected["refund"] for an outcome grader, expected["forbidden_tools"] for a trajectory grader, expected["rubric_notes"] for a judge.
Run
The result of executing one case once. This is the RunResult from Lesson 2 plus identifying metadata.
@dataclass
class Run:
case_id: str
trial: int
config_fingerprint: str
response: str
trace: "Trace"
final_state: dict
stopped_reason: str
latency_ms: float
cost_usd: floatGrader
A function from (Case, Run) to a Score. That’s the entire contract, and keeping it that narrow is what lets you mix deterministic checks, trajectory scorers, and LLM judges freely.
from typing import Protocol
class Grader(Protocol):
name: str
def grade(self, case: Case, run: Run) -> "Score": ...Score
A grader’s verdict. Always include a reason. A score without a reason is unreadable when you’re debugging 30 failures at midnight.
@dataclass
class Score:
grader: str
value: float # 0.0 to 1.0
passed: bool
reason: str = ""
details: dict = field(default_factory=dict)Normalize everything to 0–1 with a passed boolean. A judge’s 1–5 rating becomes (rating - 1) / 4 with a pass threshold; a binary assertion is 0 or 1.
Report
The aggregation over many runs: overall and per-grader pass rates, slices by tag, per-case details, cost, latency, and comparison against a baseline. Lesson 10 adds confidence intervals; Lesson 12 adds regression gates.
Grader Composition: Gates and Weights
Not all graders are equal. Some failures are disqualifying regardless of everything else. For Quill, refunding over the policy limit is a failure even if the customer is delighted. Model this with two kinds of graders:
- Gates: must pass, or the run fails outright
- Weighted: contribute to a quality score
@dataclass
class Scorecard:
gates: list[Grader]
weighted: list[tuple[Grader, float]]
pass_threshold: float = 0.7
def evaluate(self, case: Case, run: Run) -> dict:
gate_scores = [g.grade(case, run) for g in self.gates]
weighted_scores = [(g.grade(case, run), w) for g, w in self.weighted]
gates_ok = all(s.passed for s in gate_scores)
total_w = sum(w for _, w in weighted_scores) or 1.0
quality = sum(s.value * w for s, w in weighted_scores) / total_w
return {
"passed": gates_ok and quality >= self.pass_threshold,
"quality": quality if gates_ok else 0.0,
"scores": gate_scores + [s for s, _ in weighted_scores],
}A practical Quill scorecard:
| Grader | Type | Weight |
|---|---|---|
| No refund over policy limit | Gate | — |
| No action on wrong customer | Gate | — |
| Completed (didn’t hit max steps) | Gate | — |
| Correct final state | Weighted | 0.5 |
| Trajectory acceptable | Weighted | 0.2 |
| Response quality (judge) | Weighted | 0.3 |
The Runner
The runner executes cases against an agent factory, handling concurrency, repeated trials, timeouts, and errors. Errors are results, not crashes: an agent that throws is a failed run that should appear in the report.
import asyncio, time
class Runner:
def __init__(self, agent_factory, config, trials: int = 1, concurrency: int = 8,
timeout_s: float = 120):
self.agent_factory, self.config = agent_factory, config
self.trials, self.timeout_s = trials, timeout_s
self.sem = asyncio.Semaphore(concurrency)
async def _run_one(self, case: Case, trial: int) -> Run:
async with self.sem:
agent, state = self.agent_factory(self.config, case.initial_state)
t0 = time.perf_counter()
try:
result = await asyncio.wait_for(
asyncio.to_thread(agent.run, case.input, state), self.timeout_s)
stopped, response = result.stopped_reason, result.response
trace, final_state = result.trace, result.final_state
except Exception as e:
stopped, response = f"error:{type(e).__name__}", ""
trace, final_state = agent.tracer.trace, state.snapshot()
latency = (time.perf_counter() - t0) * 1000
return Run(case.id, trial, self.config.fingerprint(), response, trace,
final_state, stopped, latency, estimate_cost(trace, self.config.model))
async def run(self, cases: list[Case]) -> list[Run]:
jobs = [self._run_one(c, t) for c in cases for t in range(self.trials)]
return await asyncio.gather(*jobs)Two details matter:
- Fresh state per run. Each run gets its own copy of
initial_state. Shared mutable state between concurrent runs is the most common source of flaky eval results. - Rate limits. Concurrency is bounded by a semaphore. Provider rate limits will otherwise turn into a wave of errors that look like agent failures.
The Report
from collections import defaultdict
def build_report(cases: list[Case], runs: list[Run], scorecard: Scorecard) -> dict:
by_id = {c.id: c for c in cases}
rows = []
for run in runs:
case = by_id[run.case_id]
result = scorecard.evaluate(case, run)
rows.append({"case": case, "run": run, **result})
def rate(rs): return sum(r["passed"] for r in rs) / len(rs) if rs else 0.0
by_tag = defaultdict(list)
for r in rows:
for t in r["case"].tags:
by_tag[t].append(r)
by_grader = defaultdict(list)
for r in rows:
for s in r["scores"]:
by_grader[s.grader].append(s.passed)
return {
"config": runs[0].config_fingerprint,
"n_runs": len(rows),
"pass_rate": rate(rows),
"by_tag": {t: rate(rs) for t, rs in sorted(by_tag.items())},
"by_grader": {g: sum(v) / len(v) for g, v in by_grader.items()},
"mean_cost_usd": sum(r["run"].cost_usd for r in rows) / len(rows),
"p50_latency_ms": sorted(r["run"].latency_ms for r in rows)[len(rows) // 2],
"failures": [
{"case": r["case"].id, "trial": r["run"].trial,
"reasons": [s.reason for s in r["scores"] if not s.passed],
"trace_id": r["run"].trace.trace_id}
for r in rows if not r["passed"]
],
}The failures list is the most-read part of any report. Each entry links to a trace ID, so “why did case 47 fail?” is a single click, not a reproduction effort.
Build vs Buy
You don’t have to build all of this yourself. Here’s how hosted and open-source tools map onto the abstractions:
| Abstraction | Examples |
|---|---|
| Case / dataset | LangSmith datasets, Braintrust datasets, Langfuse datasets, plain JSONL in git |
| Runner | Inspect (UK AISI), OpenAI Evals, promptfoo, Braintrust Eval(), pytest |
| Grader | DeepEval, Ragas, autoevals, your own functions |
| Report | Braintrust, LangSmith experiments, Langfuse, a Markdown file posted to the PR |
The recommendation in this course: own the Case, Grader, and Scorecard definitions in your repo, because they encode your product’s definition of “correct.” Rent the runner, storage, and UI if it saves you time. If you swap vendors later, your test cases and graders shouldn’t have to change.
Key Takeaways
- Evaluate at four layers: component, step, trajectory, outcome. Outcome-only evals can’t explain regressions.
- Five abstractions cover almost everything: Case, Run, Grader, Score, Report
- Keep the grader contract narrow:
(Case, Run) → Score, always with a reason - Separate gates (disqualifying failures) from weighted quality scores
- Errors are results; give each run fresh state; bound concurrency
- Own the definitions of correctness in your repo; rent infrastructure if it helps
Exercise
Implement Case, Run, Score, Scorecard, Runner, and build_report for your agent. Write three trivial graders to prove the plumbing works:
CompletedGrader: passes ifstopped_reason == "final_answer"MaxStepsGrader: passes if the trace has at most 8 tool spansNonEmptyResponseGrader: passes if the response has at least 20 characters
Run five cases with three trials each and print the report. Next lesson, you’ll replace those five cases with a real dataset.
