Course Content
Golden Datasets and Scenario Suites
Building eval sets that actually represent production — sourcing from real traffic, synthetic generation, coverage matrices, stratification, and dataset versioning
The Dataset Is the Spec
Your eval dataset is the most important artifact in your evaluation framework, more important than any grader. Graders decide how to score; the dataset decides what counts. A brilliant LLM judge running over fifteen demo queries tells you almost nothing.
Think of the golden dataset as an executable specification of your agent’s job. If a behavior isn’t represented in the dataset, then as far as your process is concerned, the agent isn’t required to do it, and it will eventually stop doing it.
Start With a Coverage Matrix
Before writing a single case, decide what the dataset must cover. A coverage matrix crosses the dimensions that matter for your agent.
For Quill, three dimensions dominate:
Intent: duplicate charge, refund request, invoice question, plan change, cancellation, out-of-scope, abusive/adversarial
Difficulty: single-step lookup; multi-step with clear answer; ambiguous (needs clarification); conflicting data
Risk: read-only; money movement under limit; money movement over limit (must escalate); account-level change
INTENTS = ["duplicate_charge", "refund_request", "invoice_question",
"plan_change", "cancellation", "out_of_scope"]
DIFFICULTY = ["simple", "multi_step", "ambiguous", "conflicting"]
RISK = ["read_only", "refund_under_limit", "refund_over_limit", "account_change"]
# Not every combination is meaningful. Mark the ones that are, with a minimum count.
coverage_targets = {
("duplicate_charge", "simple", "refund_under_limit"): 8,
("duplicate_charge", "conflicting", "refund_under_limit"): 5,
("refund_request", "multi_step", "refund_over_limit"): 6,
("invoice_question", "simple", "read_only"): 6,
("cancellation", "ambiguous", "account_change"): 4,
# ...
}The matrix does two jobs. It tells you what to write, and later it tells you where you’re thin. A report that says “pass rate 91%” is less useful than one that says “pass rate 97% on simple cases and 64% on conflicting-data cases.” That second number is only possible if conflicting-data cases exist and are tagged.
Weight the matrix toward risk and difficulty, not frequency. Production traffic might be 60% simple invoice questions, but those are the cases least likely to regress in a dangerous way. A good eval set over-samples the hard and high-stakes cells.
Three Sources of Cases
1. Production Traffic (Best)
Real user inputs capture phrasing, typos, mixed intents, and context you’d never invent. If you have logs or traces:
- Sample stratified, not randomly. Cluster inputs by intent (embedding clustering works well), then sample from each cluster.
- Prioritize signal. Over-sample conversations with thumbs-down feedback, escalations, retries, long trajectories, and tool errors.
- Scrub PII before anything enters the dataset. Replace names, emails, and account numbers with consistent synthetic values.
- Reconstruct state. A production input is only a test case once you know what the world looked like at the time: which invoices existed, what the customer’s plan was. Capture this from the trace’s tool outputs.
def case_from_trace(trace, case_id: str, scrub) -> dict:
"""Turn a production trace into a draft case. A human still writes 'expected'."""
user_input = scrub(trace.spans[0].input)
state = {"customers": {}, "invoices": {}}
for s in trace.of_kind("tool"):
if s.name == "lookup_customer" and "id" in (s.output or {}):
c = scrub(s.output); state["customers"][c["id"]] = c
if s.name == "get_invoices":
for inv in s.output or []:
inv = scrub(inv); state["invoices"][inv["id"]] = inv
return {"id": case_id, "input": user_input, "initial_state": state,
"expected": {}, "tags": [], "source": f"prod:{trace.trace_id}"}2. Expert Authoring (Essential for Edge Cases)
Some cases are rare in production but critical: the customer with two accounts under similar emails, the refund that’s exactly at the $200 limit, the invoice that was already refunded once. Domain experts (support leads, in Quill’s case) should write these. They know the failure modes that matter and which ones cost money.
A useful prompt for the expert session: “Tell me about the last five tickets that went wrong, and the ones new support reps always get wrong.”
3. Synthetic Generation (Controlled Volume)
LLMs are good at generating varied phrasings and filling matrix cells. They’re bad at generating realistic difficulty: synthetic cases tend to be cleaner, more explicit, and easier than real ones. Use synthesis to fill coverage gaps, not as your primary source.
The pattern that works: generate the scenario and state programmatically, use the LLM only for the user’s message.
import random
def synth_duplicate_charge(rng: random.Random, llm) -> dict:
cid = f"C{rng.randint(1000, 9999)}"
amount = rng.choice([19.0, 49.0, 99.0, 149.0])
month = rng.choice(["January", "February", "March", "April"])
inv_a, inv_b = f"INV-{rng.randint(1000, 9999)}", f"INV-{rng.randint(1000, 9999)}"
state = {
"customers": {cid: {"id": cid, "email": f"user{cid}@example.com", "plan": "pro"}},
"invoices": {
inv_a: {"id": inv_a, "customer_id": cid, "amount": amount, "month": month, "status": "paid"},
inv_b: {"id": inv_b, "customer_id": cid, "amount": amount, "month": month, "status": "paid"},
},
}
persona = rng.choice(["terse", "frustrated", "polite and verbose", "non-native English speaker"])
message = llm(
f"Write a support message from a {persona} customer (email user{cid}@example.com) "
f"who was charged ${amount:.0f} twice in {month}. Don't mention invoice IDs. "
f"Output only the message."
)
return {
"input": message,
"initial_state": state,
"expected": {
"refund": {"invoice_id_in": [inv_a, inv_b], "amount": amount, "count": 1},
"forbidden_tools": [],
},
"tags": ["intent:duplicate_charge", "difficulty:simple", "risk:refund_under_limit",
f"persona:{persona}", "source:synthetic"],
}Because the state and expectation are constructed in code, they’re guaranteed consistent. The LLM can’t invent an invoice that doesn’t exist in the fixture. Always review a sample of synthetic cases by hand before adding them; generators drift into repetitive phrasings.
Writing Good Expectations
A case is only as good as what a grader can check. Weak expectations produce weak evals.
| Weak | Strong |
|---|---|
"expected_answer": "Refunded the duplicate charge" | "refund": {"invoice_id_in": ["INV-1", "INV-2"], "amount": 49.0, "count": 1} |
"should_escalate": true | "escalation": {"required": true, "priority_in": ["high", "urgent"]}, "refund": {"count": 0} |
"good_response": true | "must_mention": ["5-7 business days"], "must_not_claim": ["refund is instant"] |
Rules of thumb:
- Express outcomes as state. “A refund exists with these properties” beats “the response says it refunded.”
- Allow valid alternatives.
invoice_id_inaccepts either duplicate. Over-specified expectations create false failures, and a team that sees false failures stops trusting the eval. - Specify negatives. What must not happen is often more important than what must. “No refund” is an expectation.
- Leave room for judgment. For response quality, write
rubric_notesthat a judge reads (“should acknowledge the frustration; should not promise a timeline shorter than 5 business days”), not a reference answer to match.
Storage Format
JSONL in git is the right default. It diffs cleanly, reviews in pull requests, and every eval run can record the exact commit of the dataset it used.
{"id": "dup-0007", "input": "hey i got billed twice for march?? 49 bucks each", "initial_state": {"customers": {"C4410": {"id": "C4410", "email": "dana@example.com", "plan": "pro"}}, "invoices": {"INV-2290": {"id": "INV-2290", "customer_id": "C4410", "amount": 49.0, "month": "March", "status": "paid"}, "INV-2291": {"id": "INV-2291", "customer_id": "C4410", "amount": 49.0, "month": "March", "status": "paid"}}}, "expected": {"refund": {"invoice_id_in": ["INV-2290", "INV-2291"], "amount": 49.0, "count": 1}}, "tags": ["intent:duplicate_charge", "difficulty:simple", "risk:refund_under_limit", "source:prod"], "split": "regression"}Large fixtures (documents, database dumps) can live in a separate directory referenced by path.
Splits
Not every case runs every time. Split the dataset by purpose:
| Split | Size | When it runs | Purpose |
|---|---|---|---|
| smoke | 15–30 | Every commit, in minutes | Catch catastrophic breakage |
| regression | 150–500 | Every PR / nightly | The main quality number |
| adversarial | 50–200 | Nightly and before release | Safety and abuse (Lesson 9) |
| holdout | 10–20% of regression | Rarely, before major releases | Detect overfitting to the eval set |
The holdout deserves a word. If your team tunes prompts against the regression set for months, the agent will start fitting those specific cases. A holdout set that nobody looks at during development tells you whether improvements generalize. If regression goes up and holdout stays flat, you’re overfitting.
How Many Cases?
Fewer than you fear, more than you have. Lesson 10 covers the statistics properly, but as a rough guide: to detect a 5-point change in pass rate with reasonable confidence, you need a few hundred cases (or fewer cases with several trials each). For a single coverage-matrix cell, 5–10 cases is enough to notice a collapse, not to measure small changes.
Start with 50 good cases. A small, well-designed dataset beats a large sloppy one, because every failure in it is worth reading.
Versioning and Maintenance
A dataset is living code. Treat it that way:
- Version it. Every report records the dataset version (a git SHA or semantic version). Scores from different versions aren’t comparable.
- Never silently edit a case. If an expectation was wrong, fix it in a reviewed commit with a reason. Changing expectations to make the agent pass is the eval equivalent of deleting a failing test.
- Grow from failures. Every production incident and every bug report becomes at least one new case (Lesson 11 automates this).
- Prune duplicates. Near-identical cases inflate some cells and distort pass rates. Deduplicate periodically by embedding similarity.
- Re-audit labels. Policies change. When the refund limit moves from $200 to $250, every case that depends on it must be updated in the same change.
Key Takeaways
- The dataset is the executable spec of your agent’s job. Invest in it more than in graders.
- Start from a coverage matrix over intent, difficulty, and risk; over-sample hard and high-risk cells
- Source from production (realism), experts (edge cases), and synthesis (coverage), in that order of value
- Generate state and expectations in code; use the LLM only for natural-language variation
- Express expectations as state, allow valid alternatives, and specify negatives
- Split into smoke, regression, adversarial, and holdout; version everything
Exercise
Build a 50-case golden dataset for Quill (or your agent):
- Write a coverage matrix with at least 12 meaningful cells
- Author 15 edge cases by hand
- Generate 35 cases with a synthetic generator that builds state in code
- Tag every case with intent, difficulty, risk, and source
- Mark 15 cases as
smoke
Commit it as evals/datasets/quill-v1.jsonl. The next lesson writes the graders that check it.
