Press ESC to exit fullscreen
📖 Lesson ⏱️ 120 minutes

Evals in CI/CD and Regression Gates

Running evals on every prompt, model, and tool change — tiered suites, baselines, statistically sound gates, and a working GitHub Actions workflow

The Eval Nobody Runs

Most teams that build an eval framework eventually find it in the same state: it works, it’s useful, and someone has to remember to run it. So it runs before big launches, and in between, three prompt tweaks, a tool schema change, and a model version bump go out unmeasured.

An eval only protects you if it runs automatically on every change that can affect behavior, and if its result can block that change. This lesson wires the framework into CI.

What Counts as a Change

For normal software, a change is a code diff. For an agent, behavior changes when any of these change:

ChangeExampleOften missed?
Agent codeNew loop logic, retry handlingNo
System prompt“Be more concise”Yes: often edited outside code review
ModelVersion bump, provider switchYes: sometimes changes without a deploy
Tool definitionsRenamed parameter, new descriptionYes
Tool implementationsAPI now returns a different shapeSometimes
Retrieval / KB contentNew help articlesYes
ConfigTemperature, max stepsSometimes
The eval itselfDataset, graders, judge rubricSometimes

The architectural fix from Lesson 2 pays off here: if prompts, model names, tool schemas, and config all live as versioned files in the repo, every one of these changes is a diff, and CI can trigger on it. If the system prompt lives in a database that product managers edit in a web UI, you’ve built an unmonitored path to production.

# Paths that should trigger the agent eval workflow
on:
  pull_request:
    paths:
      - "agent/**"
      - "prompts/**"
      - "tools/**"
      - "config/agent*.yaml"
      - "evals/**"

For changes that don’t arrive as diffs (provider-side model updates, KB content changes), the nightly run against a pinned baseline catches them.

Tiered Suites

You can’t run 500 cases × 5 trials on every push. Split by when it runs (this mirrors the dataset splits from Lesson 4):

TierTriggerSuiteTrialsTarget timeBlocks
SmokeEvery push20–30 cases1under 3 minMerge
RegressionEvery PR (ready for review)150–300 cases + step suite2–3under 20 minMerge
FullNightly, release candidatesAll splits incl. adversarial3–5Hours OKRelease
Model adoptionManual, on new model versionFull + holdout5Hours OKModel switch

Speed tricks that keep the PR tier tolerable:

  • Mock tool backends for everything except a handful of live smoke tests
  • High concurrency, bounded by provider rate limits
  • Cache judge verdicts keyed on evidence hash (Lesson 6)
  • Run judges only on runs that passed gates
  • Selective re-running: if only the KB changed, run the cases tagged uses:kb first

Baselines

A gate needs something to compare against. Store a baseline report for the main branch: per-case results, aggregate metrics, the agent config fingerprint, dataset version, and judge model version.

def load_baseline(path="evals/baselines/main.json") -> dict:
    with open(path) as f:
        return json.load(f)

Refresh the baseline when a change merges to main. Store it as an artifact, in object storage, or in your eval platform.

Don’t compare across dataset versions. If a PR changes the dataset, rerun the baseline config on the new dataset in the same CI job. Otherwise you’re comparing scores on different exams.

Gate Design

A good gate has three layers, from strictest to most nuanced.

1. Hard Gates: Zero Tolerance

Any failure blocks the merge, regardless of averages:

  • Any policy gate failure on any case (refund limit, cross-account, leaks)
  • Any adversarial case that passed on baseline and now fails
  • Any case tagged critical that regresses
  • Smoke-suite pass rate below 100%, if the smoke suite is truly smoke

2. Statistical Regression Gate

For the aggregate quality score, use the paired comparison from Lesson 10. Block only when there’s evidence of a regression, not when the score wobbles.

def regression_gate(baseline: dict, candidate: dict, max_drop: float = 0.02) -> dict:
    """Fail if we're confident the pass rate dropped by more than max_drop."""
    cmp = paired_bootstrap(baseline["per_case"], candidate["per_case"])
    lo, hi = cmp["ci95"]
    if hi < -max_drop:
        verdict = "fail"            # confidently worse than tolerance
    elif lo < -max_drop:
        verdict = "warn"            # can't rule out a meaningful drop
    else:
        verdict = "pass"
    return {**cmp, "verdict": verdict}

The three-way verdict matters. “Warn” means “the suite can’t tell”. The author should look at regressed cases, rerun with more trials, or accept the risk explicitly. Treating every dip as a failure makes the gate flaky and trains people to override it. Treating no dip as a failure lets slow erosion through.

3. Slice Gates

Aggregate scores hide localized damage. Check key slices separately:

CRITICAL_SLICES = {"risk:refund_under_limit": 0.03, "risk:refund_over_limit": 0.0,
                   "intent:cancellation": 0.05}

def slice_gates(baseline, candidate) -> list[str]:
    failures = []
    for tag, tolerance in CRITICAL_SLICES.items():
        b, c = baseline["by_tag"].get(tag), candidate["by_tag"].get(tag)
        if b is not None and c is not None and c < b - tolerance:
            failures.append(f"{tag}: {b:.1%} → {c:.1%} (tolerance {tolerance:.0%})")
    return failures

4. Budget Gates

Quality isn’t the only thing that regresses:

  • Mean cost per task up more than 25%
  • p95 latency up more than 30%
  • Mean tool calls up more than 20%

These usually warn rather than block. Sometimes a cost increase is a deliberate tradeoff, and the author should say so in the PR.

The Eval Runner Script

One entry point that CI calls, which runs a tier and writes a machine-readable result and a human-readable report:

# evals/run.py
import argparse, asyncio, json, sys

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--tier", choices=["smoke", "regression", "full"], required=True)
    ap.add_argument("--trials", type=int, default=1)
    ap.add_argument("--baseline", default="evals/baselines/main.json")
    ap.add_argument("--out", default="eval-results")
    args = ap.parse_args()

    config = load_agent_config("config/agent.yaml")
    cases = load_cases("evals/datasets/quill.jsonl", splits=SPLITS[args.tier])
    runs = asyncio.run(Runner(make_quill, config, trials=args.trials).run(cases))
    report = build_report(cases, runs, QUILL_SCORECARD)

    baseline = load_baseline(args.baseline)
    gates = {
        "hard": hard_gate_failures(report, baseline),
        "regression": regression_gate(baseline, report),
        "slices": slice_gates(baseline, report),
        "budget": budget_warnings(baseline, report),
    }
    blocked = bool(gates["hard"]) or gates["regression"]["verdict"] == "fail" \
              or bool(gates["slices"])

    write_json(f"{args.out}/report.json", {"report": report, "gates": gates})
    write_text(f"{args.out}/summary.md", render_markdown(report, baseline, gates))
    sys.exit(1 if blocked else 0)

if __name__ == "__main__":
    main()

The GitHub Actions Workflow

name: agent-evals

on:
  pull_request:
    paths: ["agent/**", "prompts/**", "tools/**", "config/agent*.yaml", "evals/**"]
  schedule:
    - cron: "0 6 * * *"          # nightly full run
  workflow_dispatch:
    inputs:
      tier: { description: "smoke | regression | full", default: "full" }

concurrency:
  group: evals-${{ github.ref }}
  cancel-in-progress: true

jobs:
  smoke:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12", cache: "pip" }
      - run: pip install -r evals/requirements.txt
      - run: python evals/run.py --tier smoke --trials 1
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}

  regression:
    needs: smoke
    if: github.event_name == 'pull_request' && !github.event.pull_request.draft
    runs-on: ubuntu-latest
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12", cache: "pip" }
      - run: pip install -r evals/requirements.txt
      - name: Fetch main baseline
        run: gh run download --name eval-baseline-main --dir evals/baselines || true
        env: { GH_TOKEN: "${{ github.token }}" }
      - name: Run regression suite
        id: evals
        run: python evals/run.py --tier regression --trials 3
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
      - name: Post report to PR
        if: always()
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          header: agent-evals
          path: eval-results/summary.md
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: eval-results-${{ github.sha }}, path: eval-results/ }

  nightly:
    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    timeout-minutes: 240
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12", cache: "pip" }
      - run: pip install -r evals/requirements.txt
      - run: python evals/run.py --tier ${{ inputs.tier || 'full' }} --trials 5
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: eval-nightly-${{ github.run_id }}, path: eval-results/ }

A separate small workflow on push to main reruns the regression tier and uploads the result as the eval-baseline-main artifact.

Mark the regression job as a required status check in branch protection. Without that, the gate is advisory.

The PR Report

The Markdown summary is what reviewers actually read. Make it scannable, lead with the verdict, and link to traces:

## 🧪 Agent eval — regression tier (240 cases × 3 trials)

**Verdict: ⚠️ WARN** — can't rule out a drop larger than 2 pts

| Metric | main | this PR | Δ |
|---|---|---|---|
| Pass rate | 86.3% | 84.9% | −1.4 pts (95% CI −3.6, +0.8) |
| pass^3 | 71.2% | 68.8% | −2.4 pts |
| Refund slice | 91.0% | 91.7% | +0.7 |
| Cost / task | $0.021 | $0.019 | −9% |
| p95 latency | 11.8s | 10.9s | −8% |

**Hard gates:** ✅ none failed
**Regressed cases (7):** `dup-0031`, `plan-0012`, `cancel-0004` … [traces](#)
**Improved cases (4):** `refund-0019`, … [traces](#)

Handling Flakiness and Overrides

Flaky gates destroy trust faster than missing gates. If a gate fails, someone reruns, and it passes, people learn the gate is noise. Reduce flakiness with more trials in the PR tier, the three-way verdict, temperature 0 where the agent allows it, and mocked tools.

Overrides should be explicit and visible. Sometimes a regression is acceptable: a deliberate tradeoff, or a known-bad case being redefined. Allow an override via a PR label (eval-override) that requires a written justification in the PR description and a review from an eval owner. Log every override. If overrides become routine, the gate thresholds are wrong.

Changing the eval is a change. A PR that edits expectations, graders, or judge rubrics should be reviewed by someone who owns the eval, and should show the before/after score on the old agent so reviewers can see that the eval changed, not the agent.

Model Upgrades as Pull Requests

Treat a model version change exactly like code: a PR that changes config/agent.yaml from one model ID to another. It runs the regression tier automatically, then the model-adoption tier manually. The PR report becomes the record of why you did, or didn’t, adopt the new model. Pin model versions explicitly. A floating alias that the provider updates silently bypasses all of this.

Key Takeaways

  • Prompts, models, tool schemas, KB content, and config all change behavior. Make them versioned files so CI can see them.
  • Run tiered suites: smoke on every push, regression on every PR, full nightly, model adoption on demand
  • Gate in layers: hard gates (zero tolerance), a statistical regression gate with a pass/warn/fail verdict, slice gates, and budget warnings
  • Compare against a baseline on the same dataset version
  • Make the regression job a required check; make overrides explicit, justified, and logged
  • Treat model upgrades and eval changes as reviewed pull requests

Exercise

  1. Move your agent’s prompt, model ID, tool schemas, and config into versioned files
  2. Implement evals/run.py with smoke and regression tiers and the four gate layers
  3. Add the GitHub Actions workflow and the baseline-refresh workflow
  4. Open a PR that deliberately breaks something subtle (for example, remove “look up the invoice before refunding” from the prompt) and confirm the gate catches it and the report points to the right cases