Course Content
Reliability, Cost, and Latency
pass@k vs pass^k, variance and confidence intervals, sample sizes, and the cost and latency budgets that belong in every eval report
“It Passed” Is Not a Reliability Claim
Run a Quill case once and it passes. Run it ten times and it passes seven. Which is the truth? Both, and neither. The agent doesn’t have a pass/fail property per case; it has a probability of passing. Every eval number you report is an estimate of probabilities like that, and estimates come with uncertainty.
This lesson is the statistics you need to report agent quality honestly. It’s not much math, but skipping it is how teams ship regressions they “measured” as improvements.
pass@k vs pass^k
Run each case k times. Two very different metrics come out of that.
pass@k: the probability that at least one of k attempts succeeds. Useful when a human or verifier picks the best attempt, as in code generation with tests.
pass^k (“pass hat k”): the probability that all k attempts succeed. This is the one that matters for agents serving users. Each user gets one attempt, and you need the agent to succeed for every user. pass^k was popularized by the τ-bench agent benchmark precisely because pass@1 hid how inconsistent agents were.
If a case has per-trial success probability p:
- pass@k = 1 − (1 − p)^k
- pass^k = p^k
def per_case_estimates(results: dict[str, list[bool]], k: int):
"""results: case_id -> list of n trial outcomes, n >= k. Unbiased estimators."""
from math import comb
pass_at_k, pass_hat_k = [], []
for trials in results.values():
n, c = len(trials), sum(trials)
pass_at_k.append(1 - comb(n - c, k) / comb(n, k))
pass_hat_k.append(comb(c, k) / comb(n, k))
return sum(pass_at_k) / len(pass_at_k), sum(pass_hat_k) / len(pass_hat_k)A real example of how much these diverge:
pass@1 0.84
pass@4 0.96 ← looks great
pass^4 0.61 ← what users actually experience across repeat contactsAn agent with pass@1 of 84% can have only 61% of cases that work reliably. The other cases work sometimes, which in production means some users get help and others with the identical problem don’t. Report pass^k for any user-facing agent. It’s the metric most sensitive to the inconsistency that erodes trust.
Finding Flaky Cases
Per-case pass rates across trials split the dataset into three groups:
- Always pass: stable capability
- Always fail: a consistent bug, usually easy to diagnose
- Flaky (passes sometimes): the most important group. These are where small prompt or model changes swing your aggregate score, and where users experience inconsistency.
flaky = {cid: sum(t) / len(t) for cid, t in results.items() if 0 < sum(t) < len(t)}Sort flaky cases by pass rate and read their failing traces next to their passing traces. The divergence point (the first step where the two trajectories differ) is usually the decision to fix.
Confidence Intervals
A pass rate of 86% on 50 cases is not the same claim as 86% on 500. Always report an interval. For proportions, the Wilson score interval behaves well even near 0% and 100%:
from math import sqrt
def wilson_ci(successes: int, n: int, z: float = 1.96) -> tuple[float, float]:
if n == 0:
return (0.0, 1.0)
p = successes / n
denom = 1 + z**2 / n
center = (p + z**2 / (2 * n)) / denom
half = z * sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom
return (center - half, center + half)
print(wilson_ci(43, 50)) # (0.738, 0.930)
print(wilson_ci(430, 500)) # (0.827, 0.888)43/50 is “somewhere between 74% and 93%.” That’s a wide range. If yesterday’s version scored 82% on the same 50 cases, you can’t claim today’s is better.
Multiple trials per case aren’t independent samples. Ten trials of 50 cases isn’t 500 independent observations; trials of the same case are correlated. Use a cluster bootstrap: resample cases with replacement (keeping all their trials together), recompute the metric, and take percentiles.
import random
def cluster_bootstrap_ci(results: dict[str, list[bool]], metric, n_boot=2000, seed=0):
rng = random.Random(seed)
ids = list(results)
stats = []
for _ in range(n_boot):
sample = {f"{i}_{j}": results[rng.choice(ids)] for j, i in enumerate(ids)}
stats.append(metric(sample))
stats.sort()
return stats[int(0.025 * n_boot)], stats[int(0.975 * n_boot)]
mean_pass = lambda r: sum(sum(t) / len(t) for t in r.values()) / len(r)Comparing Two Versions: Use a Paired Test
The question you usually need answered isn’t “what’s the pass rate?” but “is version B better than version A?” Comparing two independent confidence intervals is wasteful. Both versions ran on the same cases, so use that pairing.
For each case, compute the difference in per-case pass rate (B minus A). Then bootstrap the mean difference:
def paired_bootstrap(a: dict[str, list[bool]], b: dict[str, list[bool]],
n_boot=5000, seed=0) -> dict:
rng = random.Random(seed)
ids = sorted(set(a) & set(b))
diffs = [sum(b[i]) / len(b[i]) - sum(a[i]) / len(a[i]) for i in ids]
observed = sum(diffs) / len(diffs)
boots = sorted(sum(rng.choice(diffs) for _ in diffs) / len(diffs) for _ in range(n_boot))
lo, hi = boots[int(0.025 * n_boot)], boots[int(0.975 * n_boot)]
return {"delta": observed, "ci95": (lo, hi),
"verdict": "better" if lo > 0 else "worse" if hi < 0 else "inconclusive",
"improved": sum(d > 0 for d in diffs), "regressed": sum(d < 0 for d in diffs)}Report this table for every comparison:
version B vs A (200 cases × 5 trials)
Δ pass rate +3.4 pts 95% CI [+0.9, +6.0] → better
cases improved 23
cases regressed 9 ← read these before mergingAlways read the regressed cases, even when the aggregate improves. A change that fixes 23 low-risk cases and breaks 9 refund cases may not be a change you want.
For binary single-trial comparisons, McNemar’s test on the discordant pairs (cases where exactly one version passed) is the classic equivalent.
How Many Cases and Trials?
To detect a difference of size d in pass rate with reasonable power, you need roughly:
def cases_needed(p: float, d: float, z_alpha=1.96, z_beta=0.84) -> int:
"""Approximate n per version for an unpaired comparison of two proportions."""
p2 = p + d
var = p * (1 - p) + p2 * (1 - p2)
return int(((z_alpha + z_beta) ** 2 * var) / d**2) + 1
for d in (0.10, 0.05, 0.02):
print(f"detect {d:.0%} change at 85% baseline → ~{cases_needed(0.85, d)} cases")detect 10% change at 85% baseline → ~138 cases
detect 5% change at 85% baseline → ~683 cases
detect 2% change at 85% baseline → ~4716 casesPaired designs and multiple trials reduce these numbers substantially, often by half or more, because they remove between-case variance. But the lesson stands: with 50 cases, you can only detect large changes. That’s fine for a smoke suite. It isn’t fine for deciding whether to adopt a new model.
Practical guidance:
| Purpose | Cases | Trials |
|---|---|---|
| Smoke (catch breakage) | 20–30 | 1 |
| PR regression (catch 5–10 pt drops) | 150–300 | 2–3 |
| Release / model adoption decision | 300–1000 | 3–5 |
| Flakiness analysis | Any | 5–10 |
Cost: Measure It Per Task
Every eval run already has token counts on its LLM spans (Lesson 2). Turn them into dollars:
PRICES_PER_MTOK = { # illustrative; load from config, prices change
"model-large": {"input": 3.00, "output": 15.00},
"model-small": {"input": 0.80, "output": 4.00},
}
def estimate_cost(trace, default_model: str) -> float:
total = 0.0
for s in trace.of_kind("llm"):
m = s.attributes.get("gen_ai.request.model", default_model)
p = PRICES_PER_MTOK[m]
total += s.attributes.get("tokens.input", 0) / 1e6 * p["input"]
total += s.attributes.get("tokens.output", 0) / 1e6 * p["output"]
return totalReport cost as a distribution, not just a mean. Agents have long tails: most tasks take four LLM calls, but a looping run takes forty. The p95 cost and the max are where budget surprises come from.
cost / task mean $0.021 p50 $0.017 p95 $0.058 max $0.31Then look at cost per successful task: total cost divided by number of successes. A cheaper model that fails more often can be more expensive per resolved ticket once retries and escalations are counted.
Latency: End-to-End and Per Step
Measure wall-clock latency per run, and break it down by span kind:
latency / task p50 4.1s p95 11.8s
llm 71%
tool 26%
overhead 3%Latency in agents is driven mostly by the number of steps, so trajectory efficiency (Lesson 7) is also a latency metric. If p95 latency jumps after a prompt change, check whether mean tool calls went up before looking at infrastructure.
The Pareto View
Quality, cost, and latency trade off. When comparing configurations (models, prompts, max steps), plot them together and look for the Pareto frontier: configurations where nothing else is better on all three axes.
config pass^3 $/task p95 latency
large model, v4 prompt 0.78 $0.041 12.3s
small model, v4 prompt 0.71 $0.009 6.1s ← Pareto
large model, v3 prompt 0.74 $0.044 13.0s ← dominated
small model + large for refunds only
0.77 $0.016 7.4s ← ParetoThe routing configuration in the last row, a small model by default and a large one for money-moving decisions, is often where the best tradeoff lives. You can only find it if every eval report includes cost and latency next to quality.
Key Takeaways
- Agents have a probability of passing each case; one run isn’t a measurement
- Report pass^k for user-facing agents; pass@k overstates reliability
- Identify flaky cases and compare their passing and failing traces
- Always report confidence intervals; use a cluster bootstrap when you have multiple trials per case
- Compare versions with a paired test, and always read the regressed cases
- 50 cases detects only large changes; size the suite to the decision it supports
- Report cost and latency as distributions, plus cost per successful task, and look for the Pareto frontier
Exercise
- Run your regression split with 5 trials per case
- Report pass@1, pass^3, and pass^5 with cluster-bootstrap confidence intervals
- List the flaky cases and diagnose the top three by comparing passing and failing traces
- Make one change (prompt, model, or max steps), rerun, and produce the paired comparison table
- Add cost (mean, p95, per success) and latency (p50, p95) to your report
