CI/CD for Skills

Turning the Lifecycle Into Something That Runs

The previous lesson defined eight lifecycle stages a skill moves through. Left as a manual process, “review” means someone remembers to check, and “test” means someone remembers to run the eval suite before merging — which, at any real scale, means they sometimes don’t. CI/CD is what turns the lifecycle from a set of good intentions into something that runs the same way, automatically, on every change.

The Pipeline Shape

Lint → Validate → Test → Evaluate → Security Scan → Publish → Registry → Deploy

Each stage maps to something you’ve already learned to do by hand earlier in this course — this lesson is about automating it, in order, with the right stages blocking a merge and the right ones only warning.

Stage 1: Lint

Basic formatting and structure checks — is the YAML frontmatter well-formed, are required fields present, does the body follow the project’s Markdown conventions. Fast, cheap, and should fail immediately on anything a human reviewer shouldn’t have to catch by eye.

Stage 2: Validate

The skills-ref validate check from the free course’s SKILL.md Fundamentals — naming rules, required-field presence, the format-level checks that determine whether a skill will even load correctly in any client.

- name: Validate SKILL.md format
  run: skills-ref validate ./skills/config-validator

Stage 3: Test

The trigger-testing practice from the free course’s Testing Skills — the labeled should-trigger / should-not-trigger query set, run automatically rather than by hand.

Stage 4: Evaluate

The full evaluation workspace from Testing and Evaluation at Scale — with-skill vs. without-skill comparison, assertion grading, benchmark collection. This is typically the slowest and most expensive stage, since it involves actually running the skill against real prompts.

Stage 5: Security Scan

The checklist from Security and Governance — automated where it can be: a secrets scanner checking for hardcoded credentials, a check that allowed-tools hasn’t silently expanded beyond what a prior review approved, a description-length or keyword check flagging unusually broad descriptions for extra human review.

Stage 6: Publish

The skill’s artifacts — SKILL.md and its bundled files — get packaged for distribution, tagged with the version determined by the change (per the previous lesson’s semver discipline).

Stage 7: Registry

The package is pushed to wherever skills are actually discovered from at your organization — covered in full in the next lesson, Skill Registries and Marketplaces.

Stage 8: Deploy

The new version becomes live for its intended audience — which, depending on the skill’s trust level, might mean immediate availability or a staged rollout to a subset of users first.

A Real Workflow

name: skill-ci

on:
  pull_request:
    paths:
      - 'skills/**'

jobs:
  lint-and-validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Lint SKILL.md frontmatter
        run: ./scripts/lint-skills.sh
      - name: Validate against Agent Skills spec
        run: skills-ref validate ./skills/${{ github.event.pull_request.changed_skill }}

  test-triggers:
    needs: lint-and-validate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run trigger eval set
        run: ./scripts/run-trigger-evals.sh skills/${{ github.event.pull_request.changed_skill }}

  evaluate-output:
    needs: test-triggers
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run with-skill / without-skill comparison
        run: ./scripts/run-output-evals.sh skills/${{ github.event.pull_request.changed_skill }}
      - name: Upload benchmark results
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-results
          path: workspace/benchmark.json

  security-scan:
    needs: lint-and-validate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Scan for hardcoded secrets
        run: ./scripts/scan-secrets.sh skills/${{ github.event.pull_request.changed_skill }}
      - name: Flag broad descriptions for review
        run: ./scripts/check-description-scope.sh skills/${{ github.event.pull_request.changed_skill }}

  publish-and-deploy:
    needs: [evaluate-output, security-scan]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Package and tag version
        run: ./scripts/package-skill.sh
      - name: Publish to registry
        run: ./scripts/publish-to-registry.sh

This isn’t a complete, ready-to-run pipeline — the referenced scripts are placeholders for logic specific to your organization’s registry and tooling — but the shape is the point: lint and validate run first and fast, testing and evaluation run next and gate on the earlier stages passing, security scanning runs in parallel with evaluation since it doesn’t depend on it, and publish/deploy only happens once everything else has passed, and only on the main branch.

What Should Block a Merge vs. What Should Only Warn

Not every check deserves the same weight. A reasonable default:

Blocking (fails the pipeline): frontmatter validation failures, trigger test regressions, security scan hits on hardcoded secrets, benchmark regressions past an agreed threshold (say, a major latency or cost increase with no corresponding quality improvement).

Warning (visible, but doesn’t block): an unusually broad description flagged for human attention, a benchmark change within normal variance, a missing changelog entry on a patch-level change.

Treating every warning as a hard block trains people to route around the pipeline rather than fix the underlying issue; treating a real security or correctness regression as a mere warning defeats the purpose of having the pipeline at all. Where that line sits is a judgment call specific to your organization’s risk tolerance — but it should be a deliberate one, not an accident of which checks happened to get implemented as blocking first.

Summary

• CI/CD turns the eight lifecycle stages from the previous lesson into something that runs the same way automatically, rather than depending on someone remembering to do it manually
• The pipeline shape — lint, validate, test, evaluate, security scan, publish, registry, deploy — mirrors practices from earlier in this course and the free companion course, now automated
• A real GitHub Actions workflow can implement this directly: fast checks first, expensive evaluation gated behind them, security scanning in parallel, publish/deploy only on a passing main branch
• Not every check should block a merge — deciding what blocks versus warns is a deliberate risk-tolerance decision, not an afterthought

Next, you’ll build the destination this pipeline’s publish stage actually pushes to — a registry where skills can be found, installed, and trusted at organizational scale.