Course Content
Trajectory and Tool-Use Evaluation
Scoring the path, not just the destination — tool selection, argument correctness, ordering, redundancy, and efficiency against reference trajectories
Right Answer, Wrong Path
Consider three Quill runs on the same duplicate-charge case. All three end with the correct refund.
Run A
lookup_customer(email) → get_invoices(C4410) → issue_refund(INV-2291, 49) → respondRun B
lookup_customer(email) → get_invoices(C4410) → search_kb("duplicate") →
get_invoices(C4410) → search_kb("refund policy") → get_invoices(C4410) →
issue_refund(INV-2291, 49) → respondRun C
issue_refund(INV-2291, 49) → respondAn outcome grader passes all three. But B is wasteful (seven tool calls, three of them repeated), and C is alarming. It refunded without ever confirming who the customer was or that the invoice was a duplicate. It got lucky because the invoice ID happened to appear in the user’s message. On the next case, that same behavior refunds the wrong person.
Trajectory evaluation is how you see the difference. It answers: was the path acceptable, safe, and efficient?
Why Exact-Match Trajectories Fail
The naive approach compares the agent’s tool sequence to a single reference sequence. As Lesson 1 showed, there are usually several valid paths. Exact match produces false failures, and a team that sees false failures learns to ignore the eval.
Instead, grade against constraints that capture what actually matters:
| Constraint | Example for Quill |
|---|---|
| Required | get_invoices must be called before any refund |
| Forbidden | issue_refund must not be called on a read-only question |
| Ordering | lookup_customer must precede get_invoices |
| Cardinality | issue_refund at most once per case |
| Budget | At most 6 tool calls in total |
| Argument | issue_refund.invoice_id must belong to the looked-up customer |
A Constraint-Based Trajectory Grader
from dataclasses import dataclass, field
@dataclass
class TrajectorySpec:
required: list[str] = field(default_factory=list)
forbidden: list[str] = field(default_factory=list)
order: list[tuple[str, str]] = field(default_factory=list) # (before, after)
max_calls: dict[str, int] = field(default_factory=dict)
max_total: int | None = None
def check_trajectory(calls: list[tuple[str, dict]], spec: TrajectorySpec) -> list[str]:
names = [n for n, _ in calls]
violations = []
for t in spec.required:
if t not in names:
violations.append(f"missing required tool: {t}")
for t in spec.forbidden:
if t in names:
violations.append(f"called forbidden tool: {t}")
for before, after in spec.order:
if after in names:
first_after = names.index(after)
if before not in names[:first_after]:
violations.append(f"{after} called before {before}")
for t, n in spec.max_calls.items():
if names.count(t) > n:
violations.append(f"{t} called {names.count(t)} times (max {n})")
if spec.max_total is not None and len(names) > spec.max_total:
violations.append(f"{len(names)} tool calls (max {spec.max_total})")
return violations
@dataclass
class TrajectoryGrader:
default_spec: TrajectorySpec
name: str = "trajectory"
def grade(self, case, run) -> Score:
spec = TrajectorySpec(**case.expected["trajectory"]) \
if "trajectory" in case.expected else self.default_spec
violations = check_trajectory(run.trace.tool_calls(), spec)
n_checks = (len(spec.required) + len(spec.forbidden) + len(spec.order)
+ len(spec.max_calls) + (spec.max_total is not None))
value = 1.0 - len(violations) / max(n_checks, 1)
return Score(self.name, max(value, 0.0), not violations,
"; ".join(violations) or "trajectory OK")A sensible default spec for Quill, applied to every case unless overridden:
QUILL_DEFAULT = TrajectorySpec(
order=[("lookup_customer", "get_invoices"), ("get_invoices", "issue_refund")],
max_calls={"issue_refund": 1, "escalate_to_human": 1},
max_total=8,
)That default alone would fail Run C (refund before get_invoices) and flag Run B if you tightened max_total to 6.
Redundancy and Efficiency
Efficiency failures don’t break the outcome, but they cost money and latency, and they’re often an early sign that the agent is confused. Two useful metrics:
def redundancy(calls: list[tuple[str, dict]]) -> float:
"""Fraction of calls that exactly repeat an earlier call (same tool, same args)."""
seen, repeats = set(), 0
for name, args in calls:
key = (name, json.dumps(args, sort_keys=True))
repeats += key in seen
seen.add(key)
return repeats / len(calls) if calls else 0.0
def efficiency(calls, reference_len: int) -> float:
"""1.0 at or below the reference length, decaying as the agent takes more steps."""
return min(1.0, reference_len / max(len(calls), 1))Run B’s redundancy is 2/7, about 29%. Track redundancy across versions. A jump usually means the agent stopped trusting its observations, often because of a prompt change or a context-window truncation.
Step-Level Tool Metrics
The trajectory grader works on whole runs. For diagnosing why trajectories go wrong, measure individual decisions. This is the step layer from Lesson 3.
Build step cases by freezing a conversation at a decision point. You can extract them automatically from reference trajectories: every prefix of a good trajectory is a step case whose expected next action is the next step in that trajectory.
@dataclass
class StepCase:
id: str
messages: list[dict] # history up to the decision
acceptable_tools: list[str] # any of these is correct
expected_args: dict = field(default_factory=dict) # key -> exact value
arg_validators: dict = field(default_factory=dict) # key -> callable
def eval_step(case: StepCase, model_client) -> dict:
msg = model_client.complete(case.messages, TOOL_SCHEMAS)
calls = msg.get("tool_calls") or []
if not calls:
chosen, args = "(final_answer)", {}
else:
chosen, args = calls[0]["name"], calls[0]["arguments"]
tool_ok = chosen in case.acceptable_tools
arg_errors = []
if tool_ok:
for k, v in case.expected_args.items():
if args.get(k) != v:
arg_errors.append(f"{k}={args.get(k)!r}, expected {v!r}")
for k, fn in case.arg_validators.items():
if not fn(args.get(k)):
arg_errors.append(f"{k}={args.get(k)!r} failed validator")
return {"tool_correct": tool_ok, "args_correct": tool_ok and not arg_errors,
"chosen": chosen, "arg_errors": arg_errors}Aggregate into three numbers you should track on every change:
- Tool selection accuracy: chose an acceptable tool (or correctly chose to answer)
- Argument accuracy: given the right tool, were the arguments correct
- Premature-answer rate: answered without calling a tool when one was needed
The last one matters more than it sounds. “Answering from memory instead of looking it up” is one of the most common regressions after a model change. The new model is more confident and skips the lookup.
Argument Matching Modes
Exact match is right for IDs and enums. It’s wrong for free-text arguments like search_kb(query) or escalate_to_human(summary). Use the right mode per argument:
| Argument type | Matching |
|---|---|
| IDs, enums, booleans | Exact |
| Amounts, dates | Normalized exact or tolerance |
| Search queries | Validator (contains key terms), or check the retrieved result instead |
| Summaries, free text | LLM judge with a narrow rubric, or skip at step level |
Reference Trajectories with Flexible Matching
Sometimes you do want to compare against a reference path, especially for long, procedural tasks. Use flexible matching modes instead of exact equality:
def match_trajectory(actual: list[str], reference: list[str], mode: str) -> bool:
if mode == "exact":
return actual == reference
if mode == "in_order": # reference is a subsequence of actual
it = iter(actual)
return all(step in it for step in reference)
if mode == "any_order": # same multiset of required steps
return all(actual.count(s) >= reference.count(s) for s in set(reference))
if mode == "superset": # reference tools all appear, extras allowed
return set(reference) <= set(actual)
raise ValueError(mode)in_order is usually the right default. It requires the essential steps in the right sequence and tolerates extra harmless steps like a KB search. These modes mirror what LangSmith’s and other frameworks’ trajectory evaluators offer, so the concepts transfer directly.
Judging the Reasoning
Some trajectory properties can’t be checked with constraints:
- Did the agent recover sensibly after a tool error, or did it retry blindly?
- When data conflicted (two invoices, different amounts), did it notice and handle the conflict?
- Was escalating the right call given what the agent knew at that point?
A trajectory judge reads the full sequence of tool calls and observations and applies a narrow rubric. Use the same discipline as Lesson 6: one criterion, binary questions, calibrated against human labels.
Criterion: Error recovery
Q1. Did any tool call return an error? (If NO, answer N/A for Q2–Q3 and PASS.)
Q2. After the error, did the agent change its approach (different args, a different
tool, or asking the user), rather than repeating the identical call?
Q3. Did the agent avoid claiming success for the action that errored?
PASS if Q1 is NO, or both Q2 and Q3 are YES.Putting It in the Scorecard
quill_scorecard = Scorecard(
gates=[CompletedGate(), RefundLimitGate(), CorrectCustomerGate()],
weighted=[
(RefundOutcomeGrader(), 0.45),
(TrajectoryGrader(QUILL_DEFAULT), 0.20),
(RubricJudge("accuracy", ACCURACY_CRITERION, ACCURACY_RUBRIC, judge), 0.25),
(RubricJudge("clarity", CLARITY_CRITERION, CLARITY_RUBRIC, judge), 0.10),
],
)And report trajectory metrics alongside the pass rate, even when they don’t affect pass/fail:
pass rate 86.0%
tool selection acc. 94.1% (step suite, 210 decisions)
argument acc. 97.3%
premature answers 2.4%
mean tool calls 4.3 (reference 3.8)
redundancy 6.1%Key Takeaways
- Outcome-only grading misses lucky successes and wasteful paths; trajectory evaluation catches both
- Grade trajectories against constraints (required, forbidden, ordering, cardinality, budget), not one exact path
- Track redundancy and efficiency; jumps signal confusion before outcomes degrade
- Measure tool selection, argument accuracy, and premature-answer rate at the step level
- Use flexible reference matching (
in_orderby default); use calibrated judges for recovery and reasoning
Exercise
- Write a default
TrajectorySpecfor your agent and per-case overrides for 10 cases - Generate at least 50 step cases automatically from reference trajectories
- Report tool-selection accuracy, argument accuracy, and premature-answer rate
- Swap the agent’s model for a different one (or change the system prompt), and compare all trajectory metrics side by side
Write one paragraph explaining which metric moved the most and what it tells you about the change.
