Press ESC to exit fullscreen
📖 Lesson ⏱️ 120 minutes

Online Evaluation and Production Monitoring

Sampling live traces, reference-free evaluators, drift detection, user-feedback signals, and the loop that turns production failures into new test cases

Offline Evals Measure the Past

Your golden dataset is a snapshot of what you thought users would do. Production is what they actually do, and it keeps changing: new product features generate new questions, a pricing change creates a wave of plan-change requests, a marketing campaign brings in customers who phrase things differently, and a provider quietly updates a model.

Offline evaluation tells you whether a change is safe to ship. Online evaluation tells you whether what you shipped is still working. You need both, and you need a pipe between them.

What’s Different Online

OfflineOnline
InputsCurated datasetLive traffic
Expected outcomeKnown per caseUnknown
VolumeHundreds of casesThousands to millions of runs
Cost toleranceGrade everythingMust sample
LatencyMinutes are fineSome checks must be real-time
PurposeDecide whether to shipDetect problems and find new cases

The biggest difference is the second row. Production runs have no expected field. Online evaluators must be reference-free: they judge a run from the run itself.

The Online Evaluation Architecture

  production ──▶ traces ──▶ sampler ──┬──▶ real-time checks (every run)
                                      │        └─▶ alerts
                                      ├──▶ async evaluators (sampled)
                                      │        └─▶ quality dashboard
                                      └──▶ review queue (low score / flagged)
                                               └─▶ human label ──▶ golden dataset

Because Lesson 2 made every run produce a structured trace, the same trace format flows into offline and online evaluation. Many graders work in both places.

Three Tiers of Online Checks

Tier 1: Real-Time, Every Run

Cheap deterministic checks that run inline or immediately after each request. These are the graders from Lesson 5 that don’t need an expectation:

  • Stop reason (max steps, error rate)
  • Policy gates (refund limit, cross-account access)
  • Tool error rate
  • Step count and token count budgets
  • Leak patterns in the response

Policy gate violations in production should page someone, not wait for a dashboard. They’re also a sign that a pre-execution guardrail is missing. Online detection after the fact isn’t a substitute for blocking the action.

Tier 2: Async, Sampled

LLM judges and heavier analysis on a sample of traffic. Reference-free versions of your offline judges:

  • Groundedness: does the response match what the tools returned? (No expectation needed; the evidence is in the trace.)
  • Resolution: did the conversation end with the user’s issue resolved, deferred, or abandoned?
  • Tone and clarity: same rubric as offline
  • Trajectory sanity: redundancy, ordering constraints that apply to every run
import random

class OnlineEvaluator:
    def __init__(self, realtime_graders, sampled_graders, sample_rate=0.1,
                 always_sample=None):
        self.rt, self.sampled = realtime_graders, sampled_graders
        self.rate = sample_rate
        self.always = always_sample or (lambda run: False)

    def process(self, run) -> dict:
        case = Case(id=run.trace.trace_id, input=run.trace.spans[0].input,
                    initial_state={}, expected={})          # reference-free
        scores = [g.grade(case, run) for g in self.rt]
        if self.always(run) or random.random() < self.rate:
            scores += [g.grade(case, run) for g in self.sampled]
        return {"trace_id": run.trace.trace_id,
                "scores": scores,
                "flag": any(not s.passed for s in scores)}

# Always judge runs that are risky or unusual, regardless of the sample rate
always = lambda run: (any(n == "issue_refund" for n, _ in run.trace.tool_calls())
                      or run.stopped_reason != "final_answer"
                      or len(run.trace.of_kind("tool")) > 8)

Stratified sampling beats uniform sampling. A 10% uniform sample of mostly simple invoice questions spends your judge budget on the least interesting traffic. Always sample money-moving runs, errors, long trajectories, and negative-feedback conversations; sample the routine traffic at a lower rate.

Tier 3: Human Review

A review queue of flagged and low-scoring runs, plus a small random sample (to estimate what the flags are missing). Humans label them with the same rubrics as offline. These labels:

  1. Become new golden-dataset cases (the feedback loop below)
  2. Continuously recalibrate the online judges (Lesson 6). If judge-human agreement on the review queue drops, the judge needs attention.

User Signals

Users tell you how the agent is doing, mostly without meaning to.

Explicit signals: thumbs up/down, CSAT ratings, written feedback. High quality but sparse (often only a few percent of conversations), and biased toward strong reactions.

Implicit signals: far more plentiful, and noisier.

SignalLikely meaning
User rephrases the same requestAgent misunderstood
User asks for a humanAgent failed or wasn’t trusted
User abandons mid-conversationFrustration, or got what they needed elsewhere
User reopens a ticket within 48hResolution didn’t stick
Very short conversation ending in “thanks”Probably success
Customer contacts support through another channel the same dayStrong failure signal
def implicit_signals(conversation, later_events) -> dict:
    user_msgs = [m["content"] for m in conversation if m["role"] == "user"]
    return {
        "rephrase": any(similar(a, b) > 0.85 for a, b in zip(user_msgs, user_msgs[1:])),
        "asked_for_human": any(re.search(r"\b(human|agent|person|representative)\b", m, re.I)
                               for m in user_msgs),
        "reopened_48h": later_events.get("reopened_within_hours", 999) <= 48,
        "turns": len(user_msgs),
    }

No single implicit signal is reliable. Combined, and validated against human labels on a sample, they make a solid quality proxy that covers 100% of traffic for free.

Drift Detection

Drift is the gradual change that makes an agent that worked last month fail this month. Monitor three kinds.

Input drift. The distribution of user requests changes. Track intent distribution (from a classifier or embedding clusters) week over week. A new cluster appearing usually means a product change generated questions your dataset doesn’t cover.

Behavior drift. The agent’s behavior changes with no code change: usually a provider-side model update, a tool API change, or a data change. Track tool-call distributions, mean steps, refusal rate, escalation rate, and response length.

Quality drift. Online judge scores and user signals trend down.

A simple, robust detector for categorical distributions is the population stability index (PSI):

from math import log

def psi(expected: dict[str, float], actual: dict[str, float], eps=1e-4) -> float:
    keys = set(expected) | set(actual)
    return sum((actual.get(k, eps) - expected.get(k, eps))
               * log(actual.get(k, eps) / expected.get(k, eps)) for k in keys)

baseline = {"duplicate_charge": .22, "refund_request": .18, "invoice_question": .41,
            "plan_change": .12, "cancellation": .07}
this_week = {"duplicate_charge": .19, "refund_request": .16, "invoice_question": .30,
             "plan_change": .28, "cancellation": .07}
print(f"PSI = {psi(baseline, this_week):.3f}")   # 0.177 → moderate shift

A common rule of thumb: PSI below 0.1 is stable, 0.1–0.25 is a moderate shift worth investigating, and above 0.25 is a significant shift. Here, plan-change requests more than doubled. Check whether the golden dataset covers plan changes well enough, and whether the agent’s pass rate on that slice holds up in production.

For continuous metrics (steps, latency, judge scores), compare rolling windows against a baseline window and alert on sustained shifts, not single spikes. Agents are noisy; alerting on every blip trains people to ignore alerts.

Alerting That People Don’t Ignore

AlertTriggerSeverity
Policy gate violationAny occurrencePage
Error / max-steps rateAbove 2x baseline for 30 minPage
Escalation rateAbove baseline + 3 SD for 2hTicket
Online groundedness7-day mean drops 5+ ptsTicket
Intent PSIAbove 0.25 week over weekTicket
Cost per task p95Above 1.5x baseline for 1 dayTicket

Every alert should link to example traces. An alert that says “groundedness dropped” without three example runs gets snoozed.

Closing the Loop: Production → Golden Dataset

This is the most valuable thing online evaluation does. Every production failure is a test case you didn’t know you needed.

 flagged run ──▶ human review ──▶ confirmed failure?
                                      │ yes
                                      ▼
                           case_from_trace() + scrub PII
                                      ▼
                         human writes `expected`, tags it
                                      ▼
                     PR to evals/datasets/ (reviewed like code)
                                      ▼
                     fails on current agent? ── no ──▶ keep anyway (regression guard)
                                      │ yes
                                      ▼
                          fix agent; case now passes

Lesson 4’s case_from_trace does the heavy lifting: it reconstructs fixture state from the trace’s tool outputs, so the production failure becomes a reproducible offline case. Combined with replay mode, you can confirm the fix against the exact sequence of tool outputs that caused the original failure.

Track the loop’s health with two numbers:

  • Cases added from production per week: should be steady, not zero
  • Time from incident to regression case: should be days, not months

Privacy and Retention

Online evaluation means storing and reading production conversations. Before you build it:

  • Scrub PII before traces reach the eval pipeline or a third-party judge
  • Respect retention limits: eval copies of traces are subject to the same data-retention policy as the originals
  • Check your judge provider’s data terms before sending customer conversations to an external model
  • Limit reviewer access to what’s needed, and log who viewed what

Key Takeaways

  • Offline evals decide whether to ship; online evals tell you whether what shipped still works
  • Production runs have no expected outcome, so online graders must be reference-free
  • Run checks in three tiers: real-time deterministic, async sampled judges, human review
  • Stratify sampling toward risky and unusual runs
  • Combine explicit and implicit user signals, validated against human labels
  • Monitor input, behavior, and quality drift; alert on sustained shifts with example traces
  • The most important output is the production-to-dataset loop

Exercise

  1. Pick six graders from earlier lessons that work reference-free and wire them into an OnlineEvaluator
  2. Define a stratified sampling rule for your agent
  3. Simulate a week of traffic (or replay real traces), and compute intent PSI against a baseline week
  4. Take three flagged runs, turn them into golden-dataset cases with case_from_trace, and confirm each reproduces the failure offline