· Agentic AI · 7 min read
📋 Prerequisites
- Trajectory and Tool-Use Evaluation (previous lesson)
🎯 What You'll Learn
- Build an LLM-driven user simulator with goals, personas, and hidden information
- Define conversation-level success and turn-level metrics for multi-turn agents
- Design memory evaluations for recall, update, and forgetting
- Attribute failures to specific agents and handoffs in a multi-agent system
Single-Turn Evals Hide Conversational Failures
Everything so far assumed one user message per case. Real support conversations aren’t like that. The customer leaves out their email, the agent asks, the customer gives the wrong one, then corrects it. The customer changes their mind halfway through. The customer gets frustrated after the second clarifying question.
Failures that only show up across turns:
- Asking for information already given, two turns earlier
- Forgetting a constraint (“don’t escalate me, I’ve been waiting three days”)
- Over-asking: four clarifying questions when one would do
- Under-asking: guessing the account when the email was ambiguous
- State drift: the agent’s understanding of the problem diverges from the user’s
To evaluate these, the test needs a counterpart that talks back.
The User Simulator
A user simulator is an LLM playing the customer. It gets a goal, a persona, and hidden information it reveals only when asked. That last part is critical. If the simulator states everything upfront, you’re back to a single-turn eval.
from dataclasses import dataclass, field
@dataclass
class UserScenario:
goal: str # what the user wants, in their terms
persona: str # tone, patience, expertise
opening: str # first message
hidden_facts: dict[str, str] # revealed only if the agent asks
constraints: list[str] = field(default_factory=list)
max_turns: int = 8
SIMULATOR_PROMPT = """You are role-playing a customer contacting billing support.
Goal: {goal}
Persona: {persona}
Facts you know (reveal a fact ONLY if the agent asks for it or it is clearly relevant):
{facts}
Your constraints: {constraints}
Rules:
- Stay in character. Write short, realistic messages.
- Never volunteer facts the agent has not asked about.
- If your goal is fully achieved, reply with exactly: [DONE]
- If the agent has clearly failed or is going in circles, reply with exactly: [GIVE_UP]
"""
class UserSimulator:
def __init__(self, scenario: UserScenario, llm):
self.s, self.llm = scenario, llm
facts = "\n".join(f"- {k}: {v}" for k, v in scenario.hidden_facts.items())
self.system = SIMULATOR_PROMPT.format(goal=scenario.goal, persona=scenario.persona,
facts=facts,
constraints="; ".join(scenario.constraints) or "none")
def reply(self, transcript: list[dict]) -> str:
# Flip roles: the agent's messages are the 'user' from the simulator's view
flipped = [{"role": "user" if m["role"] == "assistant" else "assistant",
"content": m["content"]} for m in transcript]
return self.llm(system=self.system, messages=flipped)Running a conversation:
def run_conversation(agent, sim: UserSimulator, state) -> dict:
transcript = [{"role": "user", "content": sim.s.opening}]
for turn in range(sim.s.max_turns):
agent_msg = agent.respond(transcript, state) # may call tools internally
transcript.append({"role": "assistant", "content": agent_msg})
user_msg = sim.reply(transcript)
if user_msg.strip() in ("[DONE]", "[GIVE_UP]"):
return {"transcript": transcript, "end": user_msg.strip(),
"turns": turn + 1, "final_state": state.snapshot()}
transcript.append({"role": "user", "content": user_msg})
return {"transcript": transcript, "end": "max_turns",
"turns": sim.s.max_turns, "final_state": state.snapshot()}An example scenario:
UserScenario(
goal="Get refunded for a duplicate $49 March charge",
persona="Frustrated, terse, has already contacted support once",
opening="charged twice again. fix it",
hidden_facts={
"email": "dana@example.com",
"month": "March",
"previous_ticket": "Contacted support 3 days ago, no answer",
},
constraints=["Gets annoyed if asked more than two questions",
"Does not want to be transferred to a human"],
)Simulator Pitfalls
The simulator is part of the test. If it’s too cooperative, the agent looks better than it is. If it’s inconsistent (forgets its own facts, changes its goal), you get noisy, unfair failures. Validate it:
- Read 20 simulated conversations by hand before trusting the setup
- Check that the simulator never reveals hidden facts unprompted (a simple grader can check this)
- Run each scenario several times. Simulator variance stacks on top of agent variance, so you need more trials than for single-turn cases (Lesson 10).
Don’t let the simulator decide success. [DONE] is a useful stopping signal, but the simulator can be fooled by a confident agent just like a real user. Grade success from state and a separate judge, not from the simulator’s own verdict.
Conversation-Level Metrics
| Metric | How to measure |
|---|---|
| Goal success | State assertion (refund exists) plus judge on the final resolution |
| Turns to resolution | Count turns until success; compare to a reference |
| Clarification efficiency | Number of agent questions vs number of hidden facts actually needed |
| Redundant asks | Agent asked for a fact already provided earlier in the conversation |
| Constraint adherence | Judge checks each scenario constraint against the transcript |
| Abandonment rate | Fraction ending in [GIVE_UP] or max turns |
Redundant asks are checkable without a judge if you tag facts:
def redundant_asks(transcript: list[dict], fact_patterns: dict[str, str]) -> list[str]:
"""fact_patterns: fact name -> regex that detects the agent asking for it."""
import re
provided, redundant = set(), []
for m in transcript:
if m["role"] == "user":
for fact, _ in fact_patterns.items():
if fact_in_message(fact, m["content"]):
provided.add(fact)
else:
for fact, pattern in fact_patterns.items():
if fact in provided and re.search(pattern, m["content"], re.I):
redundant.append(fact)
return redundantEvaluating Memory
Agents with memory across sessions (user preferences, prior tickets, account notes) need their own tests. Three properties matter:
Recall. Information stored in session 1 is used correctly in session 2.
Update. When information changes (“I moved to the annual plan”), the agent uses the new value, not the old one.
Forgetting and isolation. Information that should be deleted is gone; one user’s memory never appears in another user’s session.
Memory tests are multi-session scenarios:
memory_case = {
"sessions": [
{"user": "Please always email receipts to billing@acme.example, not my personal address."},
{"user": "Can you resend my April receipt?",
"expect_tool_args": {"send_receipt": {"to": "billing@acme.example"}}},
{"user": "Actually, send receipts to finance@acme.example from now on."},
{"user": "Resend May's receipt please.",
"expect_tool_args": {"send_receipt": {"to": "finance@acme.example"}}},
]
}Isolation tests are safety tests. Seed memory for user A, run a session as user B, and assert that none of A’s data appears in B’s tool calls or responses. This belongs in the adversarial suite (Lesson 9) as a gate.
For long single conversations, also test context-window pressure: pad the conversation with many turns, then check whether an early constraint is still honored. Agents that summarize or truncate history often lose exactly the constraint that mattered.
Evaluating Multi-Agent Systems
In a supervisor/worker design, a request flows through several agents. Imagine Quill v2:
┌─────────────┐
user ─────▶ │ Supervisor │ ─────▶ response
└──┬───────┬──┘
route │ │ route
┌─────▼──┐ ┌──▼──────────┐
│Billing │ │ Policy/KB │
│ worker │ │ worker │
└────────┘ └─────────────┘A wrong final answer could come from a bad routing decision, a worker error, a lossy handoff, or the supervisor misreading a worker’s result. End-to-end evaluation tells you that it failed. You need attribution to know where.
Evaluate Each Role, Then the System
- Routing accuracy. Step cases for the supervisor: given this request, which worker should handle it?
- Worker quality. Each worker evaluated in isolation with its own dataset, as a single agent.
- Handoff fidelity. Does the message the supervisor sends a worker contain everything the worker needs? A judge comparing the original request to the handoff message works well.
- Synthesis. Given correct worker outputs, does the supervisor compose a correct final response?
- End to end. The full system on the golden dataset.
Attribution With Traces
Because each agent’s activity is a subtree of spans (Lesson 2), you can attribute failures automatically: walk the trace and run the per-role graders on each subtree.
def attribute_failure(trace, case) -> str:
route = first_span(trace, kind="agent", name="supervisor.route")
if route.output["worker"] not in case.expected["acceptable_workers"]:
return "routing"
worker_root = first_span(trace, kind="agent", name=route.output["worker"])
if not worker_graders_pass(worker_root, case):
return f"worker:{route.output['worker']}"
if not handoff_complete(route.input, worker_root.input):
return "handoff"
return "synthesis"Report failures by attribution bucket:
end-to-end pass rate 81%
failures by stage:
routing 31%
worker:billing 22%
handoff 38%
synthesis 9%A breakdown like that changes what you work on next. Here, improving handoff messages is worth more than improving the billing worker.
Multi-Agent Specific Failure Modes
- Ping-pong: agents hand a task back and forth without progress. Grade with a max-handoffs budget.
- Duplicated work: two workers both perform the same action (two refunds!). The no-collateral-changes and cardinality gates from earlier lessons catch this.
- Authority confusion: a worker takes an action only the supervisor should approve. Check permissions per agent in the trace.
Key Takeaways
- Many agent failures only appear across turns; single-turn evals can’t see them
- A user simulator needs a goal, persona, and hidden facts revealed only when asked
- Validate the simulator, and never let it be the judge of success; grade from state and a separate judge
- Track turns to resolution, clarification efficiency, redundant asks, constraint adherence, and abandonment
- Memory evals test recall, update, and isolation; isolation is a safety gate
- In multi-agent systems, evaluate each role, then attribute failures using trace subtrees
Exercise
- Build a user simulator and write 10 scenarios with hidden facts and constraints for your agent
- Run each scenario 5 times; report goal success, mean turns, redundant asks, and abandonment
- Read five transcripts and note any simulator misbehavior; fix the simulator prompt
- If you have a multi-agent system, implement
attribute_failureand report failures by stage. If not, write a two-session memory test with an update step.
