Course Content
Agent Evaluation & Architecture
Almost every team building agents goes through the same moment. The demo works. A stakeholder tries it, it works for them too, and it ships. Then a model upgrade lands, or someone “improves” the system prompt, or a tool’s API returns a slightly different shape, and the agent quietly gets worse. Nobody notices for two weeks, because nobody was measuring.
The teams that get past this moment don’t have better prompts. They have an evaluation framework: a dataset that represents real work, graders they trust, a number that moves when the agent gets better or worse, and a gate that stops a regression before it reaches users.
This course teaches you to build that framework, and to design agents so they can be measured in the first place. Those two things are tied together. An agent nobody instrumented can’t be evaluated beyond its final answer, and the final answer is the least informative thing an agent produces.
The Running Example: Quill
Every lesson builds on the same system. Quill is a billing-support agent for a fictional SaaS company. A customer writes in, and Quill can:
lookup_customer(email): find the accountget_invoices(customer_id): list recent invoicessearch_kb(query): search the help centerissue_refund(invoice_id, amount, reason): refund up to $200 without approvalescalate_to_human(summary, priority): hand off to a support rep
That’s small enough to understand in one sitting, and realistic enough to have every failure mode that matters. It can refund the wrong invoice. It can refund more than policy allows. It can answer correctly by luck after calling four tools it didn’t need. It can be talked into a refund by a cleverly written ticket. And it can drift when the underlying model changes. By the capstone you’ll have a complete evaluation framework for Quill, and a template you can lift into your own agents.
What You’ll Build
- A traced, replayable agent architecture, where every LLM call, tool call, and decision is a structured span you can grade
- A golden dataset with a coverage matrix, versioned and stratified by intent, difficulty, and risk
- A grader library: deterministic assertions, state checks, trajectory scorers, and a calibrated LLM judge
- A reliability report with pass^k, confidence intervals, and cost and latency per task
- A safety suite: prompt injection through tool outputs, policy violations, and permission-boundary tests
- An online eval pipeline that samples production traces and turns failures into new test cases
- A CI regression gate, a GitHub Actions workflow that blocks a pull request when the agent gets measurably worse
How This Course Is Organized
Lessons 1–3: Foundations. Why agents need a different evaluation discipline, how to architect an agent so it can be evaluated, and the layered framework design that the rest of the course fills in.
Lessons 4–7: The core toolkit. Datasets, deterministic graders, LLM judges, and trajectory evaluation. These four lessons are the parts you’ll use every week.
Lessons 8–10: The hard cases. Multi-turn and multi-agent systems, adversarial and safety evaluation, and the statistics of reliability, cost, and latency.
Lessons 11–12: Closing the loop. Evaluating in production, and wiring evals into CI so regressions get caught at pull-request time instead of by customers.
Capstone. You build the complete framework for Quill (or your own agent) and ship it with a CI gate.
Framework-Agnostic by Design
The code in this course is plain Python with minimal dependencies. The concepts map directly onto LangSmith, Langfuse, Braintrust, Arize Phoenix, OpenAI Evals, Inspect, promptfoo, and the OpenTelemetry GenAI conventions, and lessons point out where each tool fits. Tools change every quarter. The architecture of a good eval framework hasn’t changed in years, and that’s what this course teaches.
Prerequisites, Honestly
You should already have built a tool-using agent. It doesn’t need to be in production, but you should know what a tool call looks like and have watched an agent go wrong. If you haven’t, Agentic AI Foundations covers it. This course moves fast and assumes you’re comfortable reading Python.
Let’s start with the question that motivates everything else: why can’t you just evaluate an agent the way you evaluate a model?
📋 Prerequisites
- You have built at least one tool-using LLM agent (any framework, or raw API calls)
- Comfortable reading and writing Python, including basic async and dataclasses
- Agentic AI Foundations or Production Agent Skills Engineering, or equivalent hands-on experience
🎯 What You'll Learn
- Explain why agent evaluation differs from model evaluation, and quantify how per-step errors compound across a trajectory
- Architect an agent with tracing, structured events, and replayable tools so it can be evaluated at all
- Design a layered evaluation framework covering component, step, trajectory, and outcome levels
- Build and version golden datasets that represent production traffic and edge cases
- Write deterministic graders and state-based assertions that carry most of the evaluation weight
- Design rubric-based LLM judges and calibrate them against human labels
- Evaluate tool selection, arguments, ordering, and efficiency against reference trajectories
- Evaluate multi-turn conversations, memory, and multi-agent systems with simulated users
- Build a safety and adversarial eval suite covering prompt injection and policy violations
- Report reliability with pass^k, confidence intervals, and cost and latency budgets
- Run online evaluation on production traces and feed failures back into the offline suite
- Gate prompt, model, and tool changes in CI with statistically sound regression checks

