Testing and Evaluation at Scale

Beyond “It Triggered Correctly”

The free course’s Testing Skills lesson covered whether a skill activates correctly — a small labeled query set of should-trigger and should-not-trigger prompts. That’s necessary but answers only one question. A skill can trigger perfectly and still produce a mediocre result, cost far more than it should, or behave inconsistently run to run. This lesson is the practice that answers those questions: a structured evaluation workspace, and a set of named benchmarks to run a skill against before calling it production-ready.

Designing Test Cases

A test case has three parts: a realistic prompt, a human-readable description of what success looks like, and optionally the input files the skill needs to work with.

{
  "skill_name": "csv-analyzer",
  "evals": [
    {
      "id": 1,
      "prompt": "I have a CSV of monthly sales data in data/sales_2025.csv. Can you find the top 3 months by revenue and make a bar chart?",
      "expected_output": "A bar chart image showing the top 3 months by revenue, with labeled axes and values.",
      "files": ["evals/files/sales_2025.csv"]
    }
  ]
}

Start with 2-3 test cases rather than trying to cover everything up front — you often don’t know what’s worth testing until you’ve seen a first round of real output. Vary phrasing and formality across cases, cover at least one genuine edge case (malformed input, an ambiguous request), and use realistic context — file paths, column names, backstory — the same realism discipline from the free course’s trigger-testing lesson, now applied to output quality instead of activation.

With-Skill vs. Without-Skill: The Core Pattern

Run every test case twice: once with the skill installed, once without it (or against the previous version, when you’re revising an existing skill). Without a baseline, “the output looked fine” tells you nothing about whether the skill actually contributed anything.

csv-analyzer-workspace/
└── iteration-1/
    ├── eval-top-months-chart/
    │   ├── with_skill/
    │   │   ├── outputs/       # Files produced by the run
    │   │   ├── timing.json    # Tokens and duration
    │   │   └── grading.json   # Assertion results
    │   └── without_skill/
    │       ├── outputs/
    │       ├── timing.json
    │       └── grading.json
    └── benchmark.json         # Aggregated statistics

Each run needs a clean context — no leftover state from a previous run or from the skill’s own development process, so the agent follows only what SKILL.md says. Provide, for each run: the skill path (or none, for baseline), the test prompt, any input files, and an output directory. When revising an existing skill rather than testing a new one, snapshot the previous version and use it as the baseline instead of “no skill” — this measures whether a change was actually an improvement, which is a different and often more important question than whether a skill helps at all.

Capturing Timing Data

Quality isn’t the only axis. A skill that improves output but triples token usage is a different trade-off than one that’s both better and cheaper — and that trade-off is invisible unless you record it:

{
  "total_tokens": 84852,
  "duration_ms": 23332
}

Writing Assertions

Assertions are verifiable statements about what output should contain or achieve, written after seeing a first round of real output — you often don’t know precisely what “good” looks like until the skill has actually run once.

Good assertions are specific and checkable: “the output file is valid JSON,” “the bar chart has labeled axes,” “the report includes at least 3 recommendations.” Weak assertions are either too vague to grade (“the output is good”) or too brittle to survive a correct-but-differently-worded output (“uses exactly the phrase Total Revenue: $X”). Not everything needs an assertion — writing style and visual design resist decomposition into pass/fail checks and are better caught by human review. Reserve assertions for what’s genuinely objective.

Benchmarks: The Numbers That Matter at Scale

Trigger accuracy and output quality tell you a skill works. Benchmarks tell you whether it works well enough to run at volume, across dimensions that don’t show up in a handful of manual test cases.

Latency. How long does the skill add to a response, end to end? A skill that improves output quality but adds several seconds to every single interaction may not be worth it for a high-frequency use case, even if it clearly is for an occasional one.

Accuracy. Against a labeled set of test cases with known-correct answers, what fraction does the skill get right? This only works when you actually have ground truth to check against — for tasks without a clean right answer, quality has to lean more on graded assertions and human review.

Hallucination. Specifically for skills involving retrieval or fact-based output: does the skill ever state something not actually supported by its source material? This needs deliberate test cases designed to surface it — ambiguous source data, questions the source material doesn’t actually answer — since a skill won’t reliably reveal this failure mode on its own in ordinary testing.

Token usage. The timing.json data from every run, aggregated: what does an average invocation cost, and how does that compare to the value it adds?

Cost. Token usage converted to actual dollars, at your production volume. A skill that’s marginally better per-call but runs thousands of times a day can be a much bigger budget line than the per-call token count alone suggests.

Reliability. Across many runs, how often does the skill fail outright — a tool call error, a malformed output, a timeout — rather than merely produce a lower-quality result? This is a different metric than accuracy: a skill can be highly accurate when it succeeds and still be unreliable if it fails outright on a meaningful fraction of runs.

Determinism. Run the identical prompt against the identical skill multiple times: how much does the output vary? Some variation is expected and even desirable (natural language framing differs); large variation in the substance of the answer — a different recommendation, a different number — is a signal the skill’s instructions are under-specified somewhere, and the fix belongs back in Skill Engineering’s failure-handling and validation practices.

Reading a Completed Iteration

A finished round of evaluation answers three separate questions, and they don’t always point the same direction:

1. Did the skill help? With-skill output against without-skill (or prior-version) output, on the same prompts.
2. What did it cost? Timing and token data, compared across the same two runs.
3. Where did it fail? Assertion results, plus a direct read of the raw outputs for anything an assertion wouldn’t catch.

A test case showing no improvement over baseline is a real finding, not a bug in the eval — it may mean the agent already handled that case well without the skill, which is a legitimate reason to narrow the skill’s scope (per Skill Quality) rather than force it to cover ground it doesn’t add value on.

Summary

• Trigger testing (free course) answers whether a skill activates correctly; this lesson’s evaluation practice answers whether it’s actually good, at what cost, and how reliably
• Every test case runs twice — with the skill and without it (or against a prior version) — so quality claims have a real baseline to stand on
• Assertions should be specific and objectively checkable; leave subjective quality (style, visual design) to human review
• Benchmarks — latency, accuracy, hallucination, token usage, cost, reliability, determinism — measure whether a skill holds up at production volume, not just in a handful of manual tries

You now know how to prove a skill works. The next lesson covers proving it’s safe — security and governance for skills that run with real access to real systems.