· Agentic AI · 8 min read
📋 Prerequisites
- Golden Datasets and Scenario Suites (previous lesson)
🎯 What You'll Learn
- Explain why deterministic graders should carry most of an evaluation’s weight
- Implement state, schema, trace, and text graders against the Case/Run contract
- Avoid the brittle-assertion traps that make teams distrust their evals
- Validate graders themselves with known-good and known-bad runs
The Grader Hierarchy
There’s a strong temptation, once LLM judges are available, to grade everything with one. Resist it. Graders sit on a spectrum:
deterministic code ──────────────────────────────▶ LLM judge ──▶ human review
cheap, fast, exact flexible ground truth
can't judge nuance noisy, biased slow, expensiveThe rule this course follows: if a property can be checked with code, check it with code. Use a judge only for what’s left. In a well-designed agent eval, deterministic graders decide most of the pass/fail outcomes, and the judge handles response quality and a few fuzzy properties.
Why this matters:
- Reproducibility. A deterministic grader gives the same verdict every time. When a score moves, it’s because the agent changed, not the grader.
- Cost. You’ll run evals thousands of times. A judge call per grader per run per trial adds up fast.
- Debuggability. “Refund amount was 98.0, expected 49.0” is actionable. “Judge gave 3/5” usually isn’t.
- Trust. Engineers believe assertions. They argue with judges.
State Graders: Check the World, Not the Words
The most valuable graders check run.final_state. This is why Lesson 2 put state behind a seam.
from dataclasses import dataclass
@dataclass
class RefundOutcomeGrader:
name: str = "refund_outcome"
def grade(self, case, run) -> Score:
exp = case.expected.get("refund")
if exp is None:
return Score(self.name, 1.0, True, "no refund expectation")
refunds = run.final_state["refunds"]
if len(refunds) != exp["count"]:
return Score(self.name, 0.0, False,
f"expected {exp['count']} refund(s), found {len(refunds)}",
{"refunds": refunds})
if exp["count"] == 0:
return Score(self.name, 1.0, True, "correctly issued no refund")
r = refunds[0]
if r["invoice_id"] not in exp["invoice_id_in"]:
return Score(self.name, 0.0, False,
f"refunded {r['invoice_id']}, expected one of {exp['invoice_id_in']}")
if abs(r["amount"] - exp["amount"]) > 0.005:
return Score(self.name, 0.0, False,
f"refund amount {r['amount']}, expected {exp['amount']}")
return Score(self.name, 1.0, True, f"refunded {r['invoice_id']} for {r['amount']}")The same pattern applies to any agent that changes things: a coding agent’s test suite passes, a file exists with the right contents, a database row has the right status, a calendar event exists at the right time. Outcome assertions on state are the closest thing agent evaluation has to ground truth.
Diff-Based State Checks
For agents with broader side effects, compare the whole state before and after, and assert that only the expected things changed.
def state_diff(before: dict, after: dict, path="") -> list[str]:
changes = []
for k in set(before) | set(after):
p = f"{path}.{k}" if path else k
b, a = before.get(k), after.get(k)
if isinstance(b, dict) and isinstance(a, dict):
changes += state_diff(b, a, p)
elif b != a:
changes.append(p)
return changes
@dataclass
class NoCollateralChangesGrader:
"""Fails if anything changed outside the paths this case allows."""
name: str = "no_collateral_changes"
def grade(self, case, run) -> Score:
allowed = set(case.expected.get("allowed_changes", ["refunds", "escalations"]))
changed = state_diff(case.initial_state, run.final_state)
unexpected = [c for c in changed if c.split(".")[0] not in allowed]
return Score(self.name, 0.0 if unexpected else 1.0, not unexpected,
f"unexpected changes: {unexpected}" if unexpected else "clean")This catches the class of bug where the agent does the right thing and something else. For Quill, that might be a refund done correctly plus a stray plan_change nobody asked for.
Policy Graders: Gates That Never Bend
Policy rules are deterministic by definition, so grade them deterministically. These are usually gates in the scorecard.
@dataclass
class RefundLimitGate:
limit: float = 200.0
name: str = "refund_limit_gate"
def grade(self, case, run) -> Score:
attempts = [s for s in run.trace.of_kind("tool") if s.name == "issue_refund"]
over = [s.input for s in attempts if s.input.get("amount", 0) > self.limit]
if over:
return Score(self.name, 0.0, False, f"attempted refund over limit: {over}")
return Score(self.name, 1.0, True, "within limit")Note this grades the attempt from the trace, not just the final state. The mock backend rejects over-limit refunds, so the state would look fine. But an agent that tries to refund $450 has a policy bug, even if a backend safeguard caught it. The safeguard might not exist in every environment.
@dataclass
class CorrectCustomerGate:
name: str = "correct_customer_gate"
def grade(self, case, run) -> Score:
allowed = set(case.expected.get("customer_ids", case.initial_state["customers"].keys()))
invoices = case.initial_state["invoices"]
for s in run.trace.of_kind("tool"):
if s.name in ("issue_refund",):
inv = invoices.get(s.input.get("invoice_id"), {})
if inv and inv["customer_id"] not in allowed:
return Score(self.name, 0.0, False,
f"acted on invoice of customer {inv['customer_id']}")
return Score(self.name, 1.0, True, "only acted on the right customer")Schema Graders: Structured Output Contracts
If the agent emits structured output (a final JSON decision, a ticket payload, tool arguments), validate it with a schema. Pydantic makes this a few lines.
from pydantic import BaseModel, ValidationError
from typing import Literal
class Resolution(BaseModel):
resolution: Literal["refunded", "escalated", "answered", "needs_info"]
invoice_id: str | None = None
customer_message: str
@dataclass
class ResolutionSchemaGrader:
name: str = "resolution_schema"
def grade(self, case, run) -> Score:
try:
res = Resolution.model_validate_json(extract_json(run.response))
except (ValidationError, ValueError) as e:
return Score(self.name, 0.0, False, f"invalid resolution JSON: {e}")
expected = case.expected.get("resolution")
if expected and res.resolution != expected:
return Score(self.name, 0.0, False,
f"resolution {res.resolution}, expected {expected}")
return Score(self.name, 1.0, True, res.resolution)Every tool’s arguments deserve the same treatment. A grader that validates every tool span’s input against that tool’s JSON schema catches malformed calls that a forgiving backend might silently accept.
Trace Graders: Completion, Errors, Budgets
@dataclass
class CompletedGate:
name: str = "completed"
def grade(self, case, run) -> Score:
ok = run.stopped_reason == "final_answer"
return Score(self.name, float(ok), ok, run.stopped_reason)
@dataclass
class ToolErrorRateGrader:
max_error_rate: float = 0.2
name: str = "tool_error_rate"
def grade(self, case, run) -> Score:
tools = run.trace.of_kind("tool")
errs = [s for s in tools
if s.error or (isinstance(s.output, dict) and "error" in s.output)]
rate = len(errs) / len(tools) if tools else 0.0
return Score(self.name, 1.0 - rate, rate <= self.max_error_rate,
f"{len(errs)}/{len(tools)} tool calls errored")
@dataclass
class StepBudgetGrader:
name: str = "step_budget"
def grade(self, case, run) -> Score:
budget = case.expected.get("max_tool_calls", 6)
n = len(run.trace.of_kind("tool"))
return Score(self.name, min(1.0, budget / max(n, 1)), n <= budget,
f"{n} tool calls, budget {budget}")Text Graders: Use Sparingly, Use Precisely
Some properties of the response are checkable with code. Check them precisely, and avoid the classic traps.
import re
@dataclass
class MustMentionGrader:
name: str = "must_mention"
def grade(self, case, run) -> Score:
text = run.response.lower()
missing = [p for p in case.expected.get("must_mention", []) if p.lower() not in text]
return Score(self.name, 0.0 if missing else 1.0, not missing,
f"missing: {missing}" if missing else "all present")
@dataclass
class NoLeakGrader:
"""The response must not expose internal IDs, other customers' data, or prompts."""
name: str = "no_leak"
patterns = [r"\bC\d{4}\b", r"system prompt", r"sk-[A-Za-z0-9]{20,}"]
def grade(self, case, run) -> Score:
hits = [p for p in self.patterns if re.search(p, run.response, re.I)]
return Score(self.name, 0.0 if hits else 1.0, not hits,
f"leak patterns: {hits}" if hits else "clean")The "refund" in response trap. Keyword presence doesn’t mean the agent did the thing, or even claimed to. “I’m afraid I can’t process a refund” contains “refund.” Check state for actions; use text checks only for properties that genuinely live in the text: a required disclosure, a forbidden phrase, a leaked identifier.
The exact-match trap. Comparing the full response to a reference string fails on any rephrasing. Never do this for free-text output. If you need semantic equivalence, that’s a judge’s job (Lesson 6).
The numeric-format trap. “$49”, “$49.00”, “49 USD”, and “forty-nine dollars” are the same claim. Normalize before comparing, or check the number in state instead.
Testing Your Graders
Graders are code, and code has bugs. A grader that always passes is worse than no grader, because it creates false confidence. Test each grader against hand-built runs:
def test_refund_outcome_grader():
case = make_case(expected={"refund": {"invoice_id_in": ["INV-1", "INV-2"],
"amount": 49.0, "count": 1}})
g = RefundOutcomeGrader()
assert g.grade(case, make_run(refunds=[{"invoice_id": "INV-2", "amount": 49.0}])).passed
assert not g.grade(case, make_run(refunds=[])).passed
assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-1", "amount": 49.0},
{"invoice_id": "INV-2", "amount": 49.0}])).passed
assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-9", "amount": 49.0}])).passed
assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-1", "amount": 98.0}])).passedA second check is a sanity baseline: run your grader stack against a deliberately broken agent (one that always escalates, or one that refunds the first invoice it sees). If the broken agent scores 70%, your graders aren’t discriminating.
Key Takeaways
- If a property can be checked with code, check it with code. Judges are for what’s left.
- State graders on the final environment are the closest thing to ground truth in agent eval
- Policy gates should read the trace, so they catch attempted violations, not just successful ones
- Schema graders enforce structured-output and tool-argument contracts
- Text checks are for properties that live in text (disclosures, leaks), not for whether an action happened
- Test graders with known-good and known-bad runs; sanity-check against a broken agent
Exercise
Implement for your agent:
- A state outcome grader driven by
case.expected - A no-collateral-changes grader
- Two policy gates that read the trace
- A schema grader for structured output or tool arguments
- Unit tests for each grader with at least three failing runs per grader
Then run your 50-case dataset from Lesson 4 with three trials and look at the failure list. You’ll likely find at least one failure that’s a dataset bug, not an agent bug. Fix it in a reviewed commit.
