Press ESC to exit fullscreen
📖 Lesson ⏱️ 90 minutes

Deterministic Graders and Assertions

Exact match, schema validation, state checks, and environment assertions — the cheap, reliable graders that should carry most of your eval weight

The Grader Hierarchy

There’s a strong temptation, once LLM judges are available, to grade everything with one. Resist it. Graders sit on a spectrum:

 deterministic code  ──────────────────────────────▶  LLM judge  ──▶  human review
 cheap, fast, exact                                   flexible         ground truth
 can't judge nuance                                   noisy, biased    slow, expensive

The rule this course follows: if a property can be checked with code, check it with code. Use a judge only for what’s left. In a well-designed agent eval, deterministic graders decide most of the pass/fail outcomes, and the judge handles response quality and a few fuzzy properties.

Why this matters:

  • Reproducibility. A deterministic grader gives the same verdict every time. When a score moves, it’s because the agent changed, not the grader.
  • Cost. You’ll run evals thousands of times. A judge call per grader per run per trial adds up fast.
  • Debuggability. “Refund amount was 98.0, expected 49.0” is actionable. “Judge gave 3/5” usually isn’t.
  • Trust. Engineers believe assertions. They argue with judges.

State Graders: Check the World, Not the Words

The most valuable graders check run.final_state. This is why Lesson 2 put state behind a seam.

from dataclasses import dataclass

@dataclass
class RefundOutcomeGrader:
    name: str = "refund_outcome"

    def grade(self, case, run) -> Score:
        exp = case.expected.get("refund")
        if exp is None:
            return Score(self.name, 1.0, True, "no refund expectation")

        refunds = run.final_state["refunds"]
        if len(refunds) != exp["count"]:
            return Score(self.name, 0.0, False,
                         f"expected {exp['count']} refund(s), found {len(refunds)}",
                         {"refunds": refunds})
        if exp["count"] == 0:
            return Score(self.name, 1.0, True, "correctly issued no refund")

        r = refunds[0]
        if r["invoice_id"] not in exp["invoice_id_in"]:
            return Score(self.name, 0.0, False,
                         f"refunded {r['invoice_id']}, expected one of {exp['invoice_id_in']}")
        if abs(r["amount"] - exp["amount"]) > 0.005:
            return Score(self.name, 0.0, False,
                         f"refund amount {r['amount']}, expected {exp['amount']}")
        return Score(self.name, 1.0, True, f"refunded {r['invoice_id']} for {r['amount']}")

The same pattern applies to any agent that changes things: a coding agent’s test suite passes, a file exists with the right contents, a database row has the right status, a calendar event exists at the right time. Outcome assertions on state are the closest thing agent evaluation has to ground truth.

Diff-Based State Checks

For agents with broader side effects, compare the whole state before and after, and assert that only the expected things changed.

def state_diff(before: dict, after: dict, path="") -> list[str]:
    changes = []
    for k in set(before) | set(after):
        p = f"{path}.{k}" if path else k
        b, a = before.get(k), after.get(k)
        if isinstance(b, dict) and isinstance(a, dict):
            changes += state_diff(b, a, p)
        elif b != a:
            changes.append(p)
    return changes

@dataclass
class NoCollateralChangesGrader:
    """Fails if anything changed outside the paths this case allows."""
    name: str = "no_collateral_changes"

    def grade(self, case, run) -> Score:
        allowed = set(case.expected.get("allowed_changes", ["refunds", "escalations"]))
        changed = state_diff(case.initial_state, run.final_state)
        unexpected = [c for c in changed if c.split(".")[0] not in allowed]
        return Score(self.name, 0.0 if unexpected else 1.0, not unexpected,
                     f"unexpected changes: {unexpected}" if unexpected else "clean")

This catches the class of bug where the agent does the right thing and something else. For Quill, that might be a refund done correctly plus a stray plan_change nobody asked for.

Policy Graders: Gates That Never Bend

Policy rules are deterministic by definition, so grade them deterministically. These are usually gates in the scorecard.

@dataclass
class RefundLimitGate:
    limit: float = 200.0
    name: str = "refund_limit_gate"

    def grade(self, case, run) -> Score:
        attempts = [s for s in run.trace.of_kind("tool") if s.name == "issue_refund"]
        over = [s.input for s in attempts if s.input.get("amount", 0) > self.limit]
        if over:
            return Score(self.name, 0.0, False, f"attempted refund over limit: {over}")
        return Score(self.name, 1.0, True, "within limit")

Note this grades the attempt from the trace, not just the final state. The mock backend rejects over-limit refunds, so the state would look fine. But an agent that tries to refund $450 has a policy bug, even if a backend safeguard caught it. The safeguard might not exist in every environment.

@dataclass
class CorrectCustomerGate:
    name: str = "correct_customer_gate"

    def grade(self, case, run) -> Score:
        allowed = set(case.expected.get("customer_ids", case.initial_state["customers"].keys()))
        invoices = case.initial_state["invoices"]
        for s in run.trace.of_kind("tool"):
            if s.name in ("issue_refund",):
                inv = invoices.get(s.input.get("invoice_id"), {})
                if inv and inv["customer_id"] not in allowed:
                    return Score(self.name, 0.0, False,
                                 f"acted on invoice of customer {inv['customer_id']}")
        return Score(self.name, 1.0, True, "only acted on the right customer")

Schema Graders: Structured Output Contracts

If the agent emits structured output (a final JSON decision, a ticket payload, tool arguments), validate it with a schema. Pydantic makes this a few lines.

from pydantic import BaseModel, ValidationError
from typing import Literal

class Resolution(BaseModel):
    resolution: Literal["refunded", "escalated", "answered", "needs_info"]
    invoice_id: str | None = None
    customer_message: str

@dataclass
class ResolutionSchemaGrader:
    name: str = "resolution_schema"

    def grade(self, case, run) -> Score:
        try:
            res = Resolution.model_validate_json(extract_json(run.response))
        except (ValidationError, ValueError) as e:
            return Score(self.name, 0.0, False, f"invalid resolution JSON: {e}")
        expected = case.expected.get("resolution")
        if expected and res.resolution != expected:
            return Score(self.name, 0.0, False,
                         f"resolution {res.resolution}, expected {expected}")
        return Score(self.name, 1.0, True, res.resolution)

Every tool’s arguments deserve the same treatment. A grader that validates every tool span’s input against that tool’s JSON schema catches malformed calls that a forgiving backend might silently accept.

Trace Graders: Completion, Errors, Budgets

@dataclass
class CompletedGate:
    name: str = "completed"
    def grade(self, case, run) -> Score:
        ok = run.stopped_reason == "final_answer"
        return Score(self.name, float(ok), ok, run.stopped_reason)

@dataclass
class ToolErrorRateGrader:
    max_error_rate: float = 0.2
    name: str = "tool_error_rate"
    def grade(self, case, run) -> Score:
        tools = run.trace.of_kind("tool")
        errs = [s for s in tools
                if s.error or (isinstance(s.output, dict) and "error" in s.output)]
        rate = len(errs) / len(tools) if tools else 0.0
        return Score(self.name, 1.0 - rate, rate <= self.max_error_rate,
                     f"{len(errs)}/{len(tools)} tool calls errored")

@dataclass
class StepBudgetGrader:
    name: str = "step_budget"
    def grade(self, case, run) -> Score:
        budget = case.expected.get("max_tool_calls", 6)
        n = len(run.trace.of_kind("tool"))
        return Score(self.name, min(1.0, budget / max(n, 1)), n <= budget,
                     f"{n} tool calls, budget {budget}")

Text Graders: Use Sparingly, Use Precisely

Some properties of the response are checkable with code. Check them precisely, and avoid the classic traps.

import re

@dataclass
class MustMentionGrader:
    name: str = "must_mention"
    def grade(self, case, run) -> Score:
        text = run.response.lower()
        missing = [p for p in case.expected.get("must_mention", []) if p.lower() not in text]
        return Score(self.name, 0.0 if missing else 1.0, not missing,
                     f"missing: {missing}" if missing else "all present")

@dataclass
class NoLeakGrader:
    """The response must not expose internal IDs, other customers' data, or prompts."""
    name: str = "no_leak"
    patterns = [r"\bC\d{4}\b", r"system prompt", r"sk-[A-Za-z0-9]{20,}"]
    def grade(self, case, run) -> Score:
        hits = [p for p in self.patterns if re.search(p, run.response, re.I)]
        return Score(self.name, 0.0 if hits else 1.0, not hits,
                     f"leak patterns: {hits}" if hits else "clean")

The "refund" in response trap. Keyword presence doesn’t mean the agent did the thing, or even claimed to. “I’m afraid I can’t process a refund” contains “refund.” Check state for actions; use text checks only for properties that genuinely live in the text: a required disclosure, a forbidden phrase, a leaked identifier.

The exact-match trap. Comparing the full response to a reference string fails on any rephrasing. Never do this for free-text output. If you need semantic equivalence, that’s a judge’s job (Lesson 6).

The numeric-format trap. “$49”, “$49.00”, “49 USD”, and “forty-nine dollars” are the same claim. Normalize before comparing, or check the number in state instead.

Testing Your Graders

Graders are code, and code has bugs. A grader that always passes is worse than no grader, because it creates false confidence. Test each grader against hand-built runs:

def test_refund_outcome_grader():
    case = make_case(expected={"refund": {"invoice_id_in": ["INV-1", "INV-2"],
                                          "amount": 49.0, "count": 1}})
    g = RefundOutcomeGrader()

    assert g.grade(case, make_run(refunds=[{"invoice_id": "INV-2", "amount": 49.0}])).passed
    assert not g.grade(case, make_run(refunds=[])).passed
    assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-1", "amount": 49.0},
                                               {"invoice_id": "INV-2", "amount": 49.0}])).passed
    assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-9", "amount": 49.0}])).passed
    assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-1", "amount": 98.0}])).passed

A second check is a sanity baseline: run your grader stack against a deliberately broken agent (one that always escalates, or one that refunds the first invoice it sees). If the broken agent scores 70%, your graders aren’t discriminating.

Key Takeaways

  • If a property can be checked with code, check it with code. Judges are for what’s left.
  • State graders on the final environment are the closest thing to ground truth in agent eval
  • Policy gates should read the trace, so they catch attempted violations, not just successful ones
  • Schema graders enforce structured-output and tool-argument contracts
  • Text checks are for properties that live in text (disclosures, leaks), not for whether an action happened
  • Test graders with known-good and known-bad runs; sanity-check against a broken agent

Exercise

Implement for your agent:

  1. A state outcome grader driven by case.expected
  2. A no-collateral-changes grader
  3. Two policy gates that read the trace
  4. A schema grader for structured output or tool arguments
  5. Unit tests for each grader with at least three failing runs per grader

Then run your 50-case dataset from Lesson 4 with three trials and look at the failure list. You’ll likely find at least one failure that’s a dataset bug, not an agent bug. Fix it in a reviewed commit.