Testing Skills

“It Worked Once” Isn’t a Test

You wrote a skill, tried one prompt, and it worked. That tells you the skill can work — not that it works reliably. A skill can fail in two different directions, and neither is obvious from a single successful try:

• Under-triggering — a prompt that should activate the skill doesn’t, and the agent quietly falls back to guessing instead of using your instructions.
• Over-triggering — a prompt that has nothing to do with the skill activates it anyway, pulling irrelevant instructions into context.

Both failures are silent. Nothing errors; the agent just behaves slightly worse than it should, in a way that’s easy to miss unless you’re specifically checking for it. This lesson covers two checks, done in order: manual testing first, then a small labeled query set.

Step 1: Manual Testing

Before writing anything formal, run the skill by hand against a few real prompts and watch all three stages you learned in earlier lessons:

1. Discovery — is the skill listed as available? If not, double check its location (from the discovery lesson) and that name matches the folder, exactly as covered in SKILL.md Fundamentals.
2. Activation — does a clearly on-topic prompt actually load the skill? If your client shows verbose or debug output, confirm you can see the moment it reads SKILL.md.
3. Execution — does the agent follow the instructions correctly, including any bundled scripts or references?

Try at least three prompts by hand: one obvious, on-topic request; one on-topic request phrased differently (casual instead of formal, or vice versa); and one clearly unrelated request, to confirm the skill correctly stays out of the way. If any of the three surprises you, that’s real signal — fix it before moving on rather than writing more tests against a skill you already know misbehaves.

Step 2: A Small Trigger Query Set

Manual testing catches the obvious cases. What it won’t catch is a description that’s subtly too narrow or too broad — and the only way to find that is to test against a set of realistic prompts, not just the one or two you happened to think of.

Build a small labeled list: prompts that should trigger the skill, and prompts that shouldn’t.

[
  { "query": "roll a d20 for me", "should_trigger": true },
  { "query": "I need a random number between 1 and 20 for a game", "should_trigger": true },
  { "query": "flip a coin", "should_trigger": false },
  { "query": "pick a random item from this list: apple, banana, cherry", "should_trigger": false }
]

Aim for around 10-15 queries total, split roughly evenly between the two. A few habits make this actually useful rather than a rubber stamp:

• Vary phrasing. Mix a formal request with a casual one, a terse one with a detail-heavy one.
• Include should-trigger prompts that don’t name the skill’s domain directly. “I need a random number between 1 and 20 for a game” should trigger roll-dice even though it never says the word “dice” — this is exactly the kind of case a vague description silently fails on.
• Make should-not-trigger prompts genuine near-misses, not obviously unrelated ones. “Pick a random item from this list” shares “random” with the dice skill but needs different behavior — a much more useful test than something with zero overlap, like “what’s the weather today.”

Running It

For each query, run it through your agent with the skill installed and note whether it activated — most clients show this in verbose output, a tool-call log, or an explicit “using skill X” indicator. A query passes when a should-trigger prompt activates the skill, or a should-not-trigger prompt doesn’t.

should_trigger: true,  activated: yes  →  pass
should_trigger: true,  activated: no   →  fail (under-triggering)
should_trigger: false, activated: yes  →  fail (over-triggering)
should_trigger: false, activated: no   →  pass

When something fails, the fix is almost always in the description field, not the body: add a missing keyword or phrasing for under-triggering, narrow the wording for over-triggering. Re-run the full set after any change — a fix for one query can easily shift the outcome of another.

What This Isn’t (Yet)

This lesson is about whether a skill activates correctly — not about grading the quality of what it produces once it does, or benchmarking cost and latency across versions. That’s real, important work, but it’s a deeper practice than a beginner course needs to cover, and it’s exactly where Production Agent Skills Engineering — the next course in this path — picks up: structured output evaluation, with/without-skill comparisons, and benchmarks across accuracy, cost, and reliability.

Summary

• A skill that worked once hasn’t been tested — it’s been tried
• Manual testing covers discovery, activation, and execution for a handful of real prompts
• A small labeled query set (10-15 prompts, mixed should-trigger and should-not-trigger) catches the subtler cases manual testing misses
• Fixes almost always belong in description, not the body — and require a full re-run, since one change can shift other outcomes

Next, you’ll learn how to get a working skill in front of other people — teammates, or any other agent client.