Designing the Evaluation Framework

The four evaluation layers — component, step, trajectory, and outcome — plus the core abstractions every framework needs: Case, Run, Grader, Score, Report.

🚀 advanced
⏱️ 120 minutes
👤 Bhanu Pratap Singh

· Agentic AI · 8 min read

📋 Prerequisites

  • Eval-Driven Agent Architecture (previous lesson)

🎯 What You'll Learn

  • Describe the four evaluation layers and what each one catches
  • Implement the five core abstractions of an evaluation framework: Case, Run, Grader, Score, and Report
  • Combine multiple graders into a weighted, gated scorecard
  • Run an evaluation harness concurrently and produce a comparable report

Why a Framework, Not a Script

Your first eval is usually a script: loop over some inputs, run the agent, check something, print a percentage. That’s the right way to start. It stops scaling as soon as you want to:

  • Add a second kind of check without rewriting the loop
  • Compare two agent versions on the same cases
  • Run each case several times and report variance
  • Find out which check failed on which case, and open its trace

An evaluation framework is the small set of abstractions that make all of that routine. It doesn’t need to be big. The version we build in this lesson is about 200 lines, and it’s the backbone of everything that follows.

The Four Evaluation Layers

Agents fail at different granularities. A good framework evaluates at all four, because each layer catches failures the others miss.

 ┌────────────────────────────────────────────────────────────┐
 │ 4. OUTCOME     Did the task succeed end to end?            │  few, expensive, decisive
 ├────────────────────────────────────────────────────────────┤
 │ 3. TRAJECTORY  Was the path acceptable, safe, efficient?   │
 ├────────────────────────────────────────────────────────────┤
 │ 2. STEP        Was each individual decision correct?       │
 ├────────────────────────────────────────────────────────────┤
 │ 1. COMPONENT   Does each piece work in isolation?          │  many, cheap, diagnostic
 └────────────────────────────────────────────────────────────┘

Layer 1: Component

Test pieces in isolation: a retrieval function returns the right KB article; a tool’s argument parser handles malformed dates; the intent classifier routes “I was charged twice” to billing. These are close to ordinary unit tests. They’re fast and they localize failures precisely.

Layer 2: Step

Freeze the conversation at a specific point and ask: given this exact context, does the model make the right next decision? For Quill: given a message history where get_invoices just returned two identical charges, does the next call refund the duplicate rather than the original?

Step evals are the agent equivalent of unit tests for decisions. They’re cheap (one model call each), deterministic in their setup, and the best tool for regression-testing a specific bug fix.

step_case = {
    "messages": [...],               # history up to the decision point
    "expect_tool": "issue_refund",
    "expect_args": {"invoice_id": "INV-2291"},
}

Layer 3: Trajectory

Run the full loop and grade the path: required tools called, forbidden tools avoided, correct ordering, no redundant calls, within a step budget. Covered in depth in Lesson 7.

Layer 4: Outcome

Run the full loop and grade the end state and response. Did the right refund happen? Was the customer told accurately? This is the number stakeholders care about.

The common mistake is having only Layer 4. Your outcome score drops from 88% to 81%, and you have no idea why. With step and trajectory layers, you can see that tool-selection accuracy after get_invoices fell, which points directly at the prompt change that caused it.

The Five Core Abstractions

 Case ──▶ Runner ──▶ Run ──▶ Grader(s) ──▶ Score(s) ──▶ Report

Case

One test scenario: input, setup state, expectations, and metadata for slicing results.

from dataclasses import dataclass, field
from typing import Any

@dataclass(frozen=True)
class Case:
    id: str
    input: str
    initial_state: dict
    expected: dict = field(default_factory=dict)  # free-form, read by graders
    tags: tuple[str, ...] = ()                    # e.g. ("intent:duplicate", "risk:high")
    split: str = "regression"                     # "smoke" | "regression" | "adversarial"

expected is deliberately loose. Different graders read different keys: expected["refund"] for an outcome grader, expected["forbidden_tools"] for a trajectory grader, expected["rubric_notes"] for a judge.

Run

The result of executing one case once. This is the RunResult from Lesson 2 plus identifying metadata.

@dataclass
class Run:
    case_id: str
    trial: int
    config_fingerprint: str
    response: str
    trace: "Trace"
    final_state: dict
    stopped_reason: str
    latency_ms: float
    cost_usd: float

Grader

A function from (Case, Run) to a Score. That’s the entire contract, and keeping it that narrow is what lets you mix deterministic checks, trajectory scorers, and LLM judges freely.

from typing import Protocol

class Grader(Protocol):
    name: str
    def grade(self, case: Case, run: Run) -> "Score": ...

Score

A grader’s verdict. Always include a reason. A score without a reason is unreadable when you’re debugging 30 failures at midnight.

@dataclass
class Score:
    grader: str
    value: float            # 0.0 to 1.0
    passed: bool
    reason: str = ""
    details: dict = field(default_factory=dict)

Normalize everything to 0–1 with a passed boolean. A judge’s 1–5 rating becomes (rating - 1) / 4 with a pass threshold; a binary assertion is 0 or 1.

Report

The aggregation over many runs: overall and per-grader pass rates, slices by tag, per-case details, cost, latency, and comparison against a baseline. Lesson 10 adds confidence intervals; Lesson 12 adds regression gates.

Grader Composition: Gates and Weights

Not all graders are equal. Some failures are disqualifying regardless of everything else. For Quill, refunding over the policy limit is a failure even if the customer is delighted. Model this with two kinds of graders:

  • Gates: must pass, or the run fails outright
  • Weighted: contribute to a quality score
@dataclass
class Scorecard:
    gates: list[Grader]
    weighted: list[tuple[Grader, float]]
    pass_threshold: float = 0.7

    def evaluate(self, case: Case, run: Run) -> dict:
        gate_scores = [g.grade(case, run) for g in self.gates]
        weighted_scores = [(g.grade(case, run), w) for g, w in self.weighted]

        gates_ok = all(s.passed for s in gate_scores)
        total_w = sum(w for _, w in weighted_scores) or 1.0
        quality = sum(s.value * w for s, w in weighted_scores) / total_w

        return {
            "passed": gates_ok and quality >= self.pass_threshold,
            "quality": quality if gates_ok else 0.0,
            "scores": gate_scores + [s for s, _ in weighted_scores],
        }

A practical Quill scorecard:

GraderTypeWeight
No refund over policy limitGate—
No action on wrong customerGate—
Completed (didn’t hit max steps)Gate—
Correct final stateWeighted0.5
Trajectory acceptableWeighted0.2
Response quality (judge)Weighted0.3

The Runner

The runner executes cases against an agent factory, handling concurrency, repeated trials, timeouts, and errors. Errors are results, not crashes: an agent that throws is a failed run that should appear in the report.

import asyncio, time

class Runner:
    def __init__(self, agent_factory, config, trials: int = 1, concurrency: int = 8,
                 timeout_s: float = 120):
        self.agent_factory, self.config = agent_factory, config
        self.trials, self.timeout_s = trials, timeout_s
        self.sem = asyncio.Semaphore(concurrency)

    async def _run_one(self, case: Case, trial: int) -> Run:
        async with self.sem:
            agent, state = self.agent_factory(self.config, case.initial_state)
            t0 = time.perf_counter()
            try:
                result = await asyncio.wait_for(
                    asyncio.to_thread(agent.run, case.input, state), self.timeout_s)
                stopped, response = result.stopped_reason, result.response
                trace, final_state = result.trace, result.final_state
            except Exception as e:
                stopped, response = f"error:{type(e).__name__}", ""
                trace, final_state = agent.tracer.trace, state.snapshot()
            latency = (time.perf_counter() - t0) * 1000
            return Run(case.id, trial, self.config.fingerprint(), response, trace,
                       final_state, stopped, latency, estimate_cost(trace, self.config.model))

    async def run(self, cases: list[Case]) -> list[Run]:
        jobs = [self._run_one(c, t) for c in cases for t in range(self.trials)]
        return await asyncio.gather(*jobs)

Two details matter:

  • Fresh state per run. Each run gets its own copy of initial_state. Shared mutable state between concurrent runs is the most common source of flaky eval results.
  • Rate limits. Concurrency is bounded by a semaphore. Provider rate limits will otherwise turn into a wave of errors that look like agent failures.

The Report

from collections import defaultdict

def build_report(cases: list[Case], runs: list[Run], scorecard: Scorecard) -> dict:
    by_id = {c.id: c for c in cases}
    rows = []
    for run in runs:
        case = by_id[run.case_id]
        result = scorecard.evaluate(case, run)
        rows.append({"case": case, "run": run, **result})

    def rate(rs): return sum(r["passed"] for r in rs) / len(rs) if rs else 0.0

    by_tag = defaultdict(list)
    for r in rows:
        for t in r["case"].tags:
            by_tag[t].append(r)

    by_grader = defaultdict(list)
    for r in rows:
        for s in r["scores"]:
            by_grader[s.grader].append(s.passed)

    return {
        "config": runs[0].config_fingerprint,
        "n_runs": len(rows),
        "pass_rate": rate(rows),
        "by_tag": {t: rate(rs) for t, rs in sorted(by_tag.items())},
        "by_grader": {g: sum(v) / len(v) for g, v in by_grader.items()},
        "mean_cost_usd": sum(r["run"].cost_usd for r in rows) / len(rows),
        "p50_latency_ms": sorted(r["run"].latency_ms for r in rows)[len(rows) // 2],
        "failures": [
            {"case": r["case"].id, "trial": r["run"].trial,
             "reasons": [s.reason for s in r["scores"] if not s.passed],
             "trace_id": r["run"].trace.trace_id}
            for r in rows if not r["passed"]
        ],
    }

The failures list is the most-read part of any report. Each entry links to a trace ID, so “why did case 47 fail?” is a single click, not a reproduction effort.

Build vs Buy

You don’t have to build all of this yourself. Here’s how hosted and open-source tools map onto the abstractions:

AbstractionExamples
Case / datasetLangSmith datasets, Braintrust datasets, Langfuse datasets, plain JSONL in git
RunnerInspect (UK AISI), OpenAI Evals, promptfoo, Braintrust Eval(), pytest
GraderDeepEval, Ragas, autoevals, your own functions
ReportBraintrust, LangSmith experiments, Langfuse, a Markdown file posted to the PR

The recommendation in this course: own the Case, Grader, and Scorecard definitions in your repo, because they encode your product’s definition of “correct.” Rent the runner, storage, and UI if it saves you time. If you swap vendors later, your test cases and graders shouldn’t have to change.

Key Takeaways

  • Evaluate at four layers: component, step, trajectory, outcome. Outcome-only evals can’t explain regressions.
  • Five abstractions cover almost everything: Case, Run, Grader, Score, Report
  • Keep the grader contract narrow: (Case, Run) → Score, always with a reason
  • Separate gates (disqualifying failures) from weighted quality scores
  • Errors are results; give each run fresh state; bound concurrency
  • Own the definitions of correctness in your repo; rent infrastructure if it helps

Exercise

Implement Case, Run, Score, Scorecard, Runner, and build_report for your agent. Write three trivial graders to prove the plumbing works:

  1. CompletedGrader: passes if stopped_reason == "final_answer"
  2. MaxStepsGrader: passes if the trace has at most 8 tool spans
  3. NonEmptyResponseGrader: passes if the response has at least 20 characters

Run five cases with three trials each and print the report. Next lesson, you’ll replace those five cases with a real dataset.

Part of a structured course

Agent Evaluation & Architecture

Design and build a complete Agent Evaluation Framework — traces, golden datasets, deterministic graders, calibrated LLM judges, trajectory and safety evals, reliability metrics, production monitoring, and CI regression gates. Advanced course for AI engineers and architects shipping agents to production.

Lesson 3 of 13 ⏱ 8 weeks advanced Free

Related Tutorials

🚀advanced ⏱️ 300 minutes

Capstone: Build an Agent Evaluation Framework

Assemble every piece into one framework for a real agent — dataset, graders, judge, reliability report, CI gate, and production feedback loop.

Agentic AI6 min read
agent evaluationcapstoneproject +2
🚀advanced ⏱️ 120 minutes

Evals in CI/CD and Regression Gates

Running evals on every prompt, model, and tool change — tiered suites, baselines, statistically sound gates, and a working GitHub Actions workflow.

Agentic AI9 min read
agent evaluationci/cdgithub actions +3
🚀advanced ⏱️ 90 minutes

Deterministic Graders and Assertions

Exact match, schema validation, state checks, and environment assertions — the cheap, reliable graders that should carry most of your eval weight.

Agentic AI8 min read
agent evaluationgradersassertions +2
🚀advanced ⏱️ 120 minutes

Eval-Driven Agent Architecture

Designing an agent so it can be measured — traces, spans, structured events, replayable tool layers, and the seams where evaluators attach.

Agentic AI8 min read
agent evaluationagent architectureobservability +2