· Agentic AI · 7 min read
📋 Prerequisites
- Reliability, Cost, and Latency (previous lesson)
🎯 What You'll Learn
- Explain what online evaluation catches that offline evaluation cannot
- Build reference-free evaluators that run on sampled production traces
- Combine explicit and implicit user signals into a quality monitor
- Detect input and behavior drift and alert on meaningful changes
- Close the loop by turning production failures into offline test cases
Offline Evals Measure the Past
Your golden dataset is a snapshot of what you thought users would do. Production is what they actually do, and it keeps changing: new product features generate new questions, a pricing change creates a wave of plan-change requests, a marketing campaign brings in customers who phrase things differently, and a provider quietly updates a model.
Offline evaluation tells you whether a change is safe to ship. Online evaluation tells you whether what you shipped is still working. You need both, and you need a pipe between them.
What’s Different Online
| Offline | Online | |
|---|---|---|
| Inputs | Curated dataset | Live traffic |
| Expected outcome | Known per case | Unknown |
| Volume | Hundreds of cases | Thousands to millions of runs |
| Cost tolerance | Grade everything | Must sample |
| Latency | Minutes are fine | Some checks must be real-time |
| Purpose | Decide whether to ship | Detect problems and find new cases |
The biggest difference is the second row. Production runs have no expected field. Online evaluators must be reference-free: they judge a run from the run itself.
The Online Evaluation Architecture
production ──▶ traces ──▶ sampler ──┬──▶ real-time checks (every run)
│ └─▶ alerts
├──▶ async evaluators (sampled)
│ └─▶ quality dashboard
└──▶ review queue (low score / flagged)
└─▶ human label ──▶ golden datasetBecause Lesson 2 made every run produce a structured trace, the same trace format flows into offline and online evaluation. Many graders work in both places.
Three Tiers of Online Checks
Tier 1: Real-Time, Every Run
Cheap deterministic checks that run inline or immediately after each request. These are the graders from Lesson 5 that don’t need an expectation:
- Stop reason (max steps, error rate)
- Policy gates (refund limit, cross-account access)
- Tool error rate
- Step count and token count budgets
- Leak patterns in the response
Policy gate violations in production should page someone, not wait for a dashboard. They’re also a sign that a pre-execution guardrail is missing. Online detection after the fact isn’t a substitute for blocking the action.
Tier 2: Async, Sampled
LLM judges and heavier analysis on a sample of traffic. Reference-free versions of your offline judges:
- Groundedness: does the response match what the tools returned? (No expectation needed; the evidence is in the trace.)
- Resolution: did the conversation end with the user’s issue resolved, deferred, or abandoned?
- Tone and clarity: same rubric as offline
- Trajectory sanity: redundancy, ordering constraints that apply to every run
import random
class OnlineEvaluator:
def __init__(self, realtime_graders, sampled_graders, sample_rate=0.1,
always_sample=None):
self.rt, self.sampled = realtime_graders, sampled_graders
self.rate = sample_rate
self.always = always_sample or (lambda run: False)
def process(self, run) -> dict:
case = Case(id=run.trace.trace_id, input=run.trace.spans[0].input,
initial_state={}, expected={}) # reference-free
scores = [g.grade(case, run) for g in self.rt]
if self.always(run) or random.random() < self.rate:
scores += [g.grade(case, run) for g in self.sampled]
return {"trace_id": run.trace.trace_id,
"scores": scores,
"flag": any(not s.passed for s in scores)}
# Always judge runs that are risky or unusual, regardless of the sample rate
always = lambda run: (any(n == "issue_refund" for n, _ in run.trace.tool_calls())
or run.stopped_reason != "final_answer"
or len(run.trace.of_kind("tool")) > 8)Stratified sampling beats uniform sampling. A 10% uniform sample of mostly simple invoice questions spends your judge budget on the least interesting traffic. Always sample money-moving runs, errors, long trajectories, and negative-feedback conversations; sample the routine traffic at a lower rate.
Tier 3: Human Review
A review queue of flagged and low-scoring runs, plus a small random sample (to estimate what the flags are missing). Humans label them with the same rubrics as offline. These labels:
- Become new golden-dataset cases (the feedback loop below)
- Continuously recalibrate the online judges (Lesson 6). If judge-human agreement on the review queue drops, the judge needs attention.
User Signals
Users tell you how the agent is doing, mostly without meaning to.
Explicit signals: thumbs up/down, CSAT ratings, written feedback. High quality but sparse (often only a few percent of conversations), and biased toward strong reactions.
Implicit signals: far more plentiful, and noisier.
| Signal | Likely meaning |
|---|---|
| User rephrases the same request | Agent misunderstood |
| User asks for a human | Agent failed or wasn’t trusted |
| User abandons mid-conversation | Frustration, or got what they needed elsewhere |
| User reopens a ticket within 48h | Resolution didn’t stick |
| Very short conversation ending in “thanks” | Probably success |
| Customer contacts support through another channel the same day | Strong failure signal |
def implicit_signals(conversation, later_events) -> dict:
user_msgs = [m["content"] for m in conversation if m["role"] == "user"]
return {
"rephrase": any(similar(a, b) > 0.85 for a, b in zip(user_msgs, user_msgs[1:])),
"asked_for_human": any(re.search(r"\b(human|agent|person|representative)\b", m, re.I)
for m in user_msgs),
"reopened_48h": later_events.get("reopened_within_hours", 999) <= 48,
"turns": len(user_msgs),
}No single implicit signal is reliable. Combined, and validated against human labels on a sample, they make a solid quality proxy that covers 100% of traffic for free.
Drift Detection
Drift is the gradual change that makes an agent that worked last month fail this month. Monitor three kinds.
Input drift. The distribution of user requests changes. Track intent distribution (from a classifier or embedding clusters) week over week. A new cluster appearing usually means a product change generated questions your dataset doesn’t cover.
Behavior drift. The agent’s behavior changes with no code change: usually a provider-side model update, a tool API change, or a data change. Track tool-call distributions, mean steps, refusal rate, escalation rate, and response length.
Quality drift. Online judge scores and user signals trend down.
A simple, robust detector for categorical distributions is the population stability index (PSI):
from math import log
def psi(expected: dict[str, float], actual: dict[str, float], eps=1e-4) -> float:
keys = set(expected) | set(actual)
return sum((actual.get(k, eps) - expected.get(k, eps))
* log(actual.get(k, eps) / expected.get(k, eps)) for k in keys)
baseline = {"duplicate_charge": .22, "refund_request": .18, "invoice_question": .41,
"plan_change": .12, "cancellation": .07}
this_week = {"duplicate_charge": .19, "refund_request": .16, "invoice_question": .30,
"plan_change": .28, "cancellation": .07}
print(f"PSI = {psi(baseline, this_week):.3f}") # 0.177 → moderate shiftA common rule of thumb: PSI below 0.1 is stable, 0.1–0.25 is a moderate shift worth investigating, and above 0.25 is a significant shift. Here, plan-change requests more than doubled. Check whether the golden dataset covers plan changes well enough, and whether the agent’s pass rate on that slice holds up in production.
For continuous metrics (steps, latency, judge scores), compare rolling windows against a baseline window and alert on sustained shifts, not single spikes. Agents are noisy; alerting on every blip trains people to ignore alerts.
Alerting That People Don’t Ignore
| Alert | Trigger | Severity |
|---|---|---|
| Policy gate violation | Any occurrence | Page |
| Error / max-steps rate | Above 2x baseline for 30 min | Page |
| Escalation rate | Above baseline + 3 SD for 2h | Ticket |
| Online groundedness | 7-day mean drops 5+ pts | Ticket |
| Intent PSI | Above 0.25 week over week | Ticket |
| Cost per task p95 | Above 1.5x baseline for 1 day | Ticket |
Every alert should link to example traces. An alert that says “groundedness dropped” without three example runs gets snoozed.
Closing the Loop: Production → Golden Dataset
This is the most valuable thing online evaluation does. Every production failure is a test case you didn’t know you needed.
flagged run ──▶ human review ──▶ confirmed failure?
│ yes
▼
case_from_trace() + scrub PII
▼
human writes `expected`, tags it
▼
PR to evals/datasets/ (reviewed like code)
▼
fails on current agent? ── no ──▶ keep anyway (regression guard)
│ yes
▼
fix agent; case now passesLesson 4’s case_from_trace does the heavy lifting: it reconstructs fixture state from the trace’s tool outputs, so the production failure becomes a reproducible offline case. Combined with replay mode, you can confirm the fix against the exact sequence of tool outputs that caused the original failure.
Track the loop’s health with two numbers:
- Cases added from production per week: should be steady, not zero
- Time from incident to regression case: should be days, not months
Privacy and Retention
Online evaluation means storing and reading production conversations. Before you build it:
- Scrub PII before traces reach the eval pipeline or a third-party judge
- Respect retention limits: eval copies of traces are subject to the same data-retention policy as the originals
- Check your judge provider’s data terms before sending customer conversations to an external model
- Limit reviewer access to what’s needed, and log who viewed what
Key Takeaways
- Offline evals decide whether to ship; online evals tell you whether what shipped still works
- Production runs have no expected outcome, so online graders must be reference-free
- Run checks in three tiers: real-time deterministic, async sampled judges, human review
- Stratify sampling toward risky and unusual runs
- Combine explicit and implicit user signals, validated against human labels
- Monitor input, behavior, and quality drift; alert on sustained shifts with example traces
- The most important output is the production-to-dataset loop
Exercise
- Pick six graders from earlier lessons that work reference-free and wire them into an
OnlineEvaluator - Define a stratified sampling rule for your agent
- Simulate a week of traffic (or replay real traces), and compute intent PSI against a baseline week
- Take three flagged runs, turn them into golden-dataset cases with
case_from_trace, and confirm each reproduces the failure offline
