Deterministic Graders and Assertions

Exact match, schema validation, state checks, and environment assertions — the cheap, reliable graders that should carry most of your eval weight.

🚀 advanced
⏱️ 90 minutes
👤 Bhanu Pratap Singh

· Agentic AI · 8 min read

📋 Prerequisites

  • Golden Datasets and Scenario Suites (previous lesson)

🎯 What You'll Learn

  • Explain why deterministic graders should carry most of an evaluation’s weight
  • Implement state, schema, trace, and text graders against the Case/Run contract
  • Avoid the brittle-assertion traps that make teams distrust their evals
  • Validate graders themselves with known-good and known-bad runs

The Grader Hierarchy

There’s a strong temptation, once LLM judges are available, to grade everything with one. Resist it. Graders sit on a spectrum:

 deterministic code  ──────────────────────────────▶  LLM judge  ──▶  human review
 cheap, fast, exact                                   flexible         ground truth
 can't judge nuance                                   noisy, biased    slow, expensive

The rule this course follows: if a property can be checked with code, check it with code. Use a judge only for what’s left. In a well-designed agent eval, deterministic graders decide most of the pass/fail outcomes, and the judge handles response quality and a few fuzzy properties.

Why this matters:

  • Reproducibility. A deterministic grader gives the same verdict every time. When a score moves, it’s because the agent changed, not the grader.
  • Cost. You’ll run evals thousands of times. A judge call per grader per run per trial adds up fast.
  • Debuggability. “Refund amount was 98.0, expected 49.0” is actionable. “Judge gave 3/5” usually isn’t.
  • Trust. Engineers believe assertions. They argue with judges.

State Graders: Check the World, Not the Words

The most valuable graders check run.final_state. This is why Lesson 2 put state behind a seam.

from dataclasses import dataclass

@dataclass
class RefundOutcomeGrader:
    name: str = "refund_outcome"

    def grade(self, case, run) -> Score:
        exp = case.expected.get("refund")
        if exp is None:
            return Score(self.name, 1.0, True, "no refund expectation")

        refunds = run.final_state["refunds"]
        if len(refunds) != exp["count"]:
            return Score(self.name, 0.0, False,
                         f"expected {exp['count']} refund(s), found {len(refunds)}",
                         {"refunds": refunds})
        if exp["count"] == 0:
            return Score(self.name, 1.0, True, "correctly issued no refund")

        r = refunds[0]
        if r["invoice_id"] not in exp["invoice_id_in"]:
            return Score(self.name, 0.0, False,
                         f"refunded {r['invoice_id']}, expected one of {exp['invoice_id_in']}")
        if abs(r["amount"] - exp["amount"]) > 0.005:
            return Score(self.name, 0.0, False,
                         f"refund amount {r['amount']}, expected {exp['amount']}")
        return Score(self.name, 1.0, True, f"refunded {r['invoice_id']} for {r['amount']}")

The same pattern applies to any agent that changes things: a coding agent’s test suite passes, a file exists with the right contents, a database row has the right status, a calendar event exists at the right time. Outcome assertions on state are the closest thing agent evaluation has to ground truth.

Diff-Based State Checks

For agents with broader side effects, compare the whole state before and after, and assert that only the expected things changed.

def state_diff(before: dict, after: dict, path="") -> list[str]:
    changes = []
    for k in set(before) | set(after):
        p = f"{path}.{k}" if path else k
        b, a = before.get(k), after.get(k)
        if isinstance(b, dict) and isinstance(a, dict):
            changes += state_diff(b, a, p)
        elif b != a:
            changes.append(p)
    return changes

@dataclass
class NoCollateralChangesGrader:
    """Fails if anything changed outside the paths this case allows."""
    name: str = "no_collateral_changes"

    def grade(self, case, run) -> Score:
        allowed = set(case.expected.get("allowed_changes", ["refunds", "escalations"]))
        changed = state_diff(case.initial_state, run.final_state)
        unexpected = [c for c in changed if c.split(".")[0] not in allowed]
        return Score(self.name, 0.0 if unexpected else 1.0, not unexpected,
                     f"unexpected changes: {unexpected}" if unexpected else "clean")

This catches the class of bug where the agent does the right thing and something else. For Quill, that might be a refund done correctly plus a stray plan_change nobody asked for.

Policy Graders: Gates That Never Bend

Policy rules are deterministic by definition, so grade them deterministically. These are usually gates in the scorecard.

@dataclass
class RefundLimitGate:
    limit: float = 200.0
    name: str = "refund_limit_gate"

    def grade(self, case, run) -> Score:
        attempts = [s for s in run.trace.of_kind("tool") if s.name == "issue_refund"]
        over = [s.input for s in attempts if s.input.get("amount", 0) > self.limit]
        if over:
            return Score(self.name, 0.0, False, f"attempted refund over limit: {over}")
        return Score(self.name, 1.0, True, "within limit")

Note this grades the attempt from the trace, not just the final state. The mock backend rejects over-limit refunds, so the state would look fine. But an agent that tries to refund $450 has a policy bug, even if a backend safeguard caught it. The safeguard might not exist in every environment.

@dataclass
class CorrectCustomerGate:
    name: str = "correct_customer_gate"

    def grade(self, case, run) -> Score:
        allowed = set(case.expected.get("customer_ids", case.initial_state["customers"].keys()))
        invoices = case.initial_state["invoices"]
        for s in run.trace.of_kind("tool"):
            if s.name in ("issue_refund",):
                inv = invoices.get(s.input.get("invoice_id"), {})
                if inv and inv["customer_id"] not in allowed:
                    return Score(self.name, 0.0, False,
                                 f"acted on invoice of customer {inv['customer_id']}")
        return Score(self.name, 1.0, True, "only acted on the right customer")

Schema Graders: Structured Output Contracts

If the agent emits structured output (a final JSON decision, a ticket payload, tool arguments), validate it with a schema. Pydantic makes this a few lines.

from pydantic import BaseModel, ValidationError
from typing import Literal

class Resolution(BaseModel):
    resolution: Literal["refunded", "escalated", "answered", "needs_info"]
    invoice_id: str | None = None
    customer_message: str

@dataclass
class ResolutionSchemaGrader:
    name: str = "resolution_schema"

    def grade(self, case, run) -> Score:
        try:
            res = Resolution.model_validate_json(extract_json(run.response))
        except (ValidationError, ValueError) as e:
            return Score(self.name, 0.0, False, f"invalid resolution JSON: {e}")
        expected = case.expected.get("resolution")
        if expected and res.resolution != expected:
            return Score(self.name, 0.0, False,
                         f"resolution {res.resolution}, expected {expected}")
        return Score(self.name, 1.0, True, res.resolution)

Every tool’s arguments deserve the same treatment. A grader that validates every tool span’s input against that tool’s JSON schema catches malformed calls that a forgiving backend might silently accept.

Trace Graders: Completion, Errors, Budgets

@dataclass
class CompletedGate:
    name: str = "completed"
    def grade(self, case, run) -> Score:
        ok = run.stopped_reason == "final_answer"
        return Score(self.name, float(ok), ok, run.stopped_reason)

@dataclass
class ToolErrorRateGrader:
    max_error_rate: float = 0.2
    name: str = "tool_error_rate"
    def grade(self, case, run) -> Score:
        tools = run.trace.of_kind("tool")
        errs = [s for s in tools
                if s.error or (isinstance(s.output, dict) and "error" in s.output)]
        rate = len(errs) / len(tools) if tools else 0.0
        return Score(self.name, 1.0 - rate, rate <= self.max_error_rate,
                     f"{len(errs)}/{len(tools)} tool calls errored")

@dataclass
class StepBudgetGrader:
    name: str = "step_budget"
    def grade(self, case, run) -> Score:
        budget = case.expected.get("max_tool_calls", 6)
        n = len(run.trace.of_kind("tool"))
        return Score(self.name, min(1.0, budget / max(n, 1)), n <= budget,
                     f"{n} tool calls, budget {budget}")

Text Graders: Use Sparingly, Use Precisely

Some properties of the response are checkable with code. Check them precisely, and avoid the classic traps.

import re

@dataclass
class MustMentionGrader:
    name: str = "must_mention"
    def grade(self, case, run) -> Score:
        text = run.response.lower()
        missing = [p for p in case.expected.get("must_mention", []) if p.lower() not in text]
        return Score(self.name, 0.0 if missing else 1.0, not missing,
                     f"missing: {missing}" if missing else "all present")

@dataclass
class NoLeakGrader:
    """The response must not expose internal IDs, other customers' data, or prompts."""
    name: str = "no_leak"
    patterns = [r"\bC\d{4}\b", r"system prompt", r"sk-[A-Za-z0-9]{20,}"]
    def grade(self, case, run) -> Score:
        hits = [p for p in self.patterns if re.search(p, run.response, re.I)]
        return Score(self.name, 0.0 if hits else 1.0, not hits,
                     f"leak patterns: {hits}" if hits else "clean")

The "refund" in response trap. Keyword presence doesn’t mean the agent did the thing, or even claimed to. “I’m afraid I can’t process a refund” contains “refund.” Check state for actions; use text checks only for properties that genuinely live in the text: a required disclosure, a forbidden phrase, a leaked identifier.

The exact-match trap. Comparing the full response to a reference string fails on any rephrasing. Never do this for free-text output. If you need semantic equivalence, that’s a judge’s job (Lesson 6).

The numeric-format trap. “$49”, “$49.00”, “49 USD”, and “forty-nine dollars” are the same claim. Normalize before comparing, or check the number in state instead.

Testing Your Graders

Graders are code, and code has bugs. A grader that always passes is worse than no grader, because it creates false confidence. Test each grader against hand-built runs:

def test_refund_outcome_grader():
    case = make_case(expected={"refund": {"invoice_id_in": ["INV-1", "INV-2"],
                                          "amount": 49.0, "count": 1}})
    g = RefundOutcomeGrader()

    assert g.grade(case, make_run(refunds=[{"invoice_id": "INV-2", "amount": 49.0}])).passed
    assert not g.grade(case, make_run(refunds=[])).passed
    assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-1", "amount": 49.0},
                                               {"invoice_id": "INV-2", "amount": 49.0}])).passed
    assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-9", "amount": 49.0}])).passed
    assert not g.grade(case, make_run(refunds=[{"invoice_id": "INV-1", "amount": 98.0}])).passed

A second check is a sanity baseline: run your grader stack against a deliberately broken agent (one that always escalates, or one that refunds the first invoice it sees). If the broken agent scores 70%, your graders aren’t discriminating.

Key Takeaways

  • If a property can be checked with code, check it with code. Judges are for what’s left.
  • State graders on the final environment are the closest thing to ground truth in agent eval
  • Policy gates should read the trace, so they catch attempted violations, not just successful ones
  • Schema graders enforce structured-output and tool-argument contracts
  • Text checks are for properties that live in text (disclosures, leaks), not for whether an action happened
  • Test graders with known-good and known-bad runs; sanity-check against a broken agent

Exercise

Implement for your agent:

  1. A state outcome grader driven by case.expected
  2. A no-collateral-changes grader
  3. Two policy gates that read the trace
  4. A schema grader for structured output or tool arguments
  5. Unit tests for each grader with at least three failing runs per grader

Then run your 50-case dataset from Lesson 4 with three trials and look at the failure list. You’ll likely find at least one failure that’s a dataset bug, not an agent bug. Fix it in a reviewed commit.

Part of a structured course

Agent Evaluation & Architecture

Design and build a complete Agent Evaluation Framework — traces, golden datasets, deterministic graders, calibrated LLM judges, trajectory and safety evals, reliability metrics, production monitoring, and CI regression gates. Advanced course for AI engineers and architects shipping agents to production.

Lesson 5 of 13 ⏱ 8 weeks advanced Free

Related Tutorials

🚀advanced ⏱️ 300 minutes

Capstone: Build an Agent Evaluation Framework

Assemble every piece into one framework for a real agent — dataset, graders, judge, reliability report, CI gate, and production feedback loop.

Agentic AI6 min read
agent evaluationcapstoneproject +2
🚀advanced ⏱️ 120 minutes

Evals in CI/CD and Regression Gates

Running evals on every prompt, model, and tool change — tiered suites, baselines, statistically sound gates, and a working GitHub Actions workflow.

Agentic AI9 min read
agent evaluationci/cdgithub actions +3
🚀advanced ⏱️ 120 minutes

Eval-Driven Agent Architecture

Designing an agent so it can be measured — traces, spans, structured events, replayable tool layers, and the seams where evaluators attach.

Agentic AI8 min read
agent evaluationagent architectureobservability +2
🚀advanced ⏱️ 120 minutes

Designing the Evaluation Framework

The four evaluation layers — component, step, trajectory, and outcome — plus the core abstractions every framework needs: Case, Run, Grader, Score, Report.

Agentic AI8 min read
agent evaluationeval frameworkarchitecture +1