· Agentic AI · 9 min read
📋 Prerequisites
- Online Evaluation and Production Monitoring (previous lesson)
🎯 What You'll Learn
- Identify every kind of change that should trigger an agent evaluation
- Design tiered eval suites that fit into PR, nightly, and release pipelines
- Implement regression gates with baselines, hard gates, and statistically sound thresholds
- Build a GitHub Actions workflow that runs evals and posts a report on the pull request
The Eval Nobody Runs
Most teams that build an eval framework eventually find it in the same state: it works, it’s useful, and someone has to remember to run it. So it runs before big launches, and in between, three prompt tweaks, a tool schema change, and a model version bump go out unmeasured.
An eval only protects you if it runs automatically on every change that can affect behavior, and if its result can block that change. This lesson wires the framework into CI.
What Counts as a Change
For normal software, a change is a code diff. For an agent, behavior changes when any of these change:
| Change | Example | Often missed? |
|---|---|---|
| Agent code | New loop logic, retry handling | No |
| System prompt | “Be more concise” | Yes: often edited outside code review |
| Model | Version bump, provider switch | Yes: sometimes changes without a deploy |
| Tool definitions | Renamed parameter, new description | Yes |
| Tool implementations | API now returns a different shape | Sometimes |
| Retrieval / KB content | New help articles | Yes |
| Config | Temperature, max steps | Sometimes |
| The eval itself | Dataset, graders, judge rubric | Sometimes |
The architectural fix from Lesson 2 pays off here: if prompts, model names, tool schemas, and config all live as versioned files in the repo, every one of these changes is a diff, and CI can trigger on it. If the system prompt lives in a database that product managers edit in a web UI, you’ve built an unmonitored path to production.
# Paths that should trigger the agent eval workflow
on:
pull_request:
paths:
- "agent/**"
- "prompts/**"
- "tools/**"
- "config/agent*.yaml"
- "evals/**"For changes that don’t arrive as diffs (provider-side model updates, KB content changes), the nightly run against a pinned baseline catches them.
Tiered Suites
You can’t run 500 cases × 5 trials on every push. Split by when it runs (this mirrors the dataset splits from Lesson 4):
| Tier | Trigger | Suite | Trials | Target time | Blocks |
|---|---|---|---|---|---|
| Smoke | Every push | 20–30 cases | 1 | under 3 min | Merge |
| Regression | Every PR (ready for review) | 150–300 cases + step suite | 2–3 | under 20 min | Merge |
| Full | Nightly, release candidates | All splits incl. adversarial | 3–5 | Hours OK | Release |
| Model adoption | Manual, on new model version | Full + holdout | 5 | Hours OK | Model switch |
Speed tricks that keep the PR tier tolerable:
- Mock tool backends for everything except a handful of live smoke tests
- High concurrency, bounded by provider rate limits
- Cache judge verdicts keyed on evidence hash (Lesson 6)
- Run judges only on runs that passed gates
- Selective re-running: if only the KB changed, run the cases tagged
uses:kbfirst
Baselines
A gate needs something to compare against. Store a baseline report for the main branch: per-case results, aggregate metrics, the agent config fingerprint, dataset version, and judge model version.
def load_baseline(path="evals/baselines/main.json") -> dict:
with open(path) as f:
return json.load(f)Refresh the baseline when a change merges to main. Store it as an artifact, in object storage, or in your eval platform.
Don’t compare across dataset versions. If a PR changes the dataset, rerun the baseline config on the new dataset in the same CI job. Otherwise you’re comparing scores on different exams.
Gate Design
A good gate has three layers, from strictest to most nuanced.
1. Hard Gates: Zero Tolerance
Any failure blocks the merge, regardless of averages:
- Any policy gate failure on any case (refund limit, cross-account, leaks)
- Any adversarial case that passed on baseline and now fails
- Any case tagged
criticalthat regresses - Smoke-suite pass rate below 100%, if the smoke suite is truly smoke
2. Statistical Regression Gate
For the aggregate quality score, use the paired comparison from Lesson 10. Block only when there’s evidence of a regression, not when the score wobbles.
def regression_gate(baseline: dict, candidate: dict, max_drop: float = 0.02) -> dict:
"""Fail if we're confident the pass rate dropped by more than max_drop."""
cmp = paired_bootstrap(baseline["per_case"], candidate["per_case"])
lo, hi = cmp["ci95"]
if hi < -max_drop:
verdict = "fail" # confidently worse than tolerance
elif lo < -max_drop:
verdict = "warn" # can't rule out a meaningful drop
else:
verdict = "pass"
return {**cmp, "verdict": verdict}The three-way verdict matters. “Warn” means “the suite can’t tell”. The author should look at regressed cases, rerun with more trials, or accept the risk explicitly. Treating every dip as a failure makes the gate flaky and trains people to override it. Treating no dip as a failure lets slow erosion through.
3. Slice Gates
Aggregate scores hide localized damage. Check key slices separately:
CRITICAL_SLICES = {"risk:refund_under_limit": 0.03, "risk:refund_over_limit": 0.0,
"intent:cancellation": 0.05}
def slice_gates(baseline, candidate) -> list[str]:
failures = []
for tag, tolerance in CRITICAL_SLICES.items():
b, c = baseline["by_tag"].get(tag), candidate["by_tag"].get(tag)
if b is not None and c is not None and c < b - tolerance:
failures.append(f"{tag}: {b:.1%} → {c:.1%} (tolerance {tolerance:.0%})")
return failures4. Budget Gates
Quality isn’t the only thing that regresses:
- Mean cost per task up more than 25%
- p95 latency up more than 30%
- Mean tool calls up more than 20%
These usually warn rather than block. Sometimes a cost increase is a deliberate tradeoff, and the author should say so in the PR.
The Eval Runner Script
One entry point that CI calls, which runs a tier and writes a machine-readable result and a human-readable report:
# evals/run.py
import argparse, asyncio, json, sys
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--tier", choices=["smoke", "regression", "full"], required=True)
ap.add_argument("--trials", type=int, default=1)
ap.add_argument("--baseline", default="evals/baselines/main.json")
ap.add_argument("--out", default="eval-results")
args = ap.parse_args()
config = load_agent_config("config/agent.yaml")
cases = load_cases("evals/datasets/quill.jsonl", splits=SPLITS[args.tier])
runs = asyncio.run(Runner(make_quill, config, trials=args.trials).run(cases))
report = build_report(cases, runs, QUILL_SCORECARD)
baseline = load_baseline(args.baseline)
gates = {
"hard": hard_gate_failures(report, baseline),
"regression": regression_gate(baseline, report),
"slices": slice_gates(baseline, report),
"budget": budget_warnings(baseline, report),
}
blocked = bool(gates["hard"]) or gates["regression"]["verdict"] == "fail" \
or bool(gates["slices"])
write_json(f"{args.out}/report.json", {"report": report, "gates": gates})
write_text(f"{args.out}/summary.md", render_markdown(report, baseline, gates))
sys.exit(1 if blocked else 0)
if __name__ == "__main__":
main()The GitHub Actions Workflow
name: agent-evals
on:
pull_request:
paths: ["agent/**", "prompts/**", "tools/**", "config/agent*.yaml", "evals/**"]
schedule:
- cron: "0 6 * * *" # nightly full run
workflow_dispatch:
inputs:
tier: { description: "smoke | regression | full", default: "full" }
concurrency:
group: evals-${{ github.ref }}
cancel-in-progress: true
jobs:
smoke:
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12", cache: "pip" }
- run: pip install -r evals/requirements.txt
- run: python evals/run.py --tier smoke --trials 1
env:
LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
regression:
needs: smoke
if: github.event_name == 'pull_request' && !github.event.pull_request.draft
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12", cache: "pip" }
- run: pip install -r evals/requirements.txt
- name: Fetch main baseline
run: gh run download --name eval-baseline-main --dir evals/baselines || true
env: { GH_TOKEN: "${{ github.token }}" }
- name: Run regression suite
id: evals
run: python evals/run.py --tier regression --trials 3
env:
LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
- name: Post report to PR
if: always()
uses: marocchino/sticky-pull-request-comment@v2
with:
header: agent-evals
path: eval-results/summary.md
- uses: actions/upload-artifact@v4
if: always()
with: { name: eval-results-${{ github.sha }}, path: eval-results/ }
nightly:
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
timeout-minutes: 240
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12", cache: "pip" }
- run: pip install -r evals/requirements.txt
- run: python evals/run.py --tier ${{ inputs.tier || 'full' }} --trials 5
env:
LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
- uses: actions/upload-artifact@v4
if: always()
with: { name: eval-nightly-${{ github.run_id }}, path: eval-results/ }A separate small workflow on push to main reruns the regression tier and uploads the result as the eval-baseline-main artifact.
Mark the regression job as a required status check in branch protection. Without that, the gate is advisory.
The PR Report
The Markdown summary is what reviewers actually read. Make it scannable, lead with the verdict, and link to traces:
## 🧪 Agent eval — regression tier (240 cases × 3 trials)
**Verdict: ⚠️ WARN** — can't rule out a drop larger than 2 pts
| Metric | main | this PR | Δ |
|---|---|---|---|
| Pass rate | 86.3% | 84.9% | −1.4 pts (95% CI −3.6, +0.8) |
| pass^3 | 71.2% | 68.8% | −2.4 pts |
| Refund slice | 91.0% | 91.7% | +0.7 |
| Cost / task | $0.021 | $0.019 | −9% |
| p95 latency | 11.8s | 10.9s | −8% |
**Hard gates:** ✅ none failed
**Regressed cases (7):** `dup-0031`, `plan-0012`, `cancel-0004` … [traces](#)
**Improved cases (4):** `refund-0019`, … [traces](#)Handling Flakiness and Overrides
Flaky gates destroy trust faster than missing gates. If a gate fails, someone reruns, and it passes, people learn the gate is noise. Reduce flakiness with more trials in the PR tier, the three-way verdict, temperature 0 where the agent allows it, and mocked tools.
Overrides should be explicit and visible. Sometimes a regression is acceptable: a deliberate tradeoff, or a known-bad case being redefined. Allow an override via a PR label (eval-override) that requires a written justification in the PR description and a review from an eval owner. Log every override. If overrides become routine, the gate thresholds are wrong.
Changing the eval is a change. A PR that edits expectations, graders, or judge rubrics should be reviewed by someone who owns the eval, and should show the before/after score on the old agent so reviewers can see that the eval changed, not the agent.
Model Upgrades as Pull Requests
Treat a model version change exactly like code: a PR that changes config/agent.yaml from one model ID to another. It runs the regression tier automatically, then the model-adoption tier manually. The PR report becomes the record of why you did, or didn’t, adopt the new model. Pin model versions explicitly. A floating alias that the provider updates silently bypasses all of this.
Key Takeaways
- Prompts, models, tool schemas, KB content, and config all change behavior. Make them versioned files so CI can see them.
- Run tiered suites: smoke on every push, regression on every PR, full nightly, model adoption on demand
- Gate in layers: hard gates (zero tolerance), a statistical regression gate with a pass/warn/fail verdict, slice gates, and budget warnings
- Compare against a baseline on the same dataset version
- Make the regression job a required check; make overrides explicit, justified, and logged
- Treat model upgrades and eval changes as reviewed pull requests
Exercise
- Move your agent’s prompt, model ID, tool schemas, and config into versioned files
- Implement
evals/run.pywith smoke and regression tiers and the four gate layers - Add the GitHub Actions workflow and the baseline-refresh workflow
- Open a PR that deliberately breaks something subtle (for example, remove “look up the invoice before refunding” from the prompt) and confirm the gate catches it and the report points to the right cases
