How Teams Manage Skills Files — /skill-doctor, Git, Evals

The lifecycle, as files

Store: one repo, one skill per folder

skills/
  add-structured-logs/SKILL.md
  review-pr/SKILL.md
  cut-release/SKILL.md
  evals/
    add-structured-logs.eval.md
    review-pr.eval.md
  install.sh

Each SKILL.md starts with frontmatter the harness reads — a name and a one-line description that states when it applies, because that description is what the model sees in the listing. A description that says "use when adding logs to a Node service; not for browser code" fires more precisely than "logging helper" and costs the same.

Load: symlink per harness, never copy

#!/usr/bin/env bash
# install.sh - link every skill into each harness that reads skills
set -euo pipefail
SRC="$HOME/dev/skills"
for target in "$HOME/.claude/skills" "$HOME/.codex/skills"; do
  mkdir -p "$target"
  for d in "$SRC"/*/; do
    name=$(basename "$d")
    ln -sfn "$d" "$target/$name"
  done
done

Copies drift; links do not. Teams using Guix Home or a dotfiles manager reported the same pattern with bidirectional links. If you distribute to a whole team, a plugin marketplace with version bumps is the heavier alternative — it gives you auto-updates at the cost of a publishing step.

Verify: an eval per skill

An eval is a fixed prompt plus an assertion. The simplest useful shape is a markdown file with the task, the files it should touch, and three things the output must contain. Run it on every change to the skill and once a fortnight regardless. When an eval fails after a model update, that is drift, and the fix is usually a sentence in the skill, not a rewrite.

Measure and prune: monthly

Run /skill-doctor, delete anything never invoked, move anything rarely invoked behind the skill-finder pattern so it loads on demand, and check /cost shows the cache staying warm across turns afterwards.

Where this is heading

The thread's opener noted that model capability may eventually absorb what skills encode. The counter-view, which we hold, is that skills are less about capability and more about your taste and your constraints — the release process your company uses, the log format your dashboards parse — and no base model learns those. What will change is the loading mechanism: lazy, searched, measured. /skill-doctor is the first harness feature built on that assumption.

An eval file, concretely

The thread agreed on evals but nobody posted one, so here is the smallest shape that has caught real regressions for us. One markdown file per skill, three parts: the fixture, the prompt, the assertions.

# evals/add-structured-logs.eval.md

## Fixture
repo: fixtures/node-service   (a 40-file Express app, committed)

## Prompt
Use the add-structured-logs skill to add request logging to src/routes/orders.ts

## Must be true after the run
- src/routes/orders.ts imports the shared logger, not console
- every new log line includes requestId and route
- no other file changed
- npm test passes

A runner script copies the fixture to a temp directory, runs Claude Code headless with the prompt, then checks each assertion with grep, git diff and the test command. Green means the skill still does what it says under the current model. Red after a model or harness update is drift, and the fix is nearly always one clarifying sentence in the skill's instructions.

The fortnightly drift job

The two-week documentation diff one commenter described is a small cron. Ours is a headless session with a fixed prompt: for each skill that encodes vendor documentation, fetch the current doc, compare it to the claims in the skill, and open a pull request with the changed lines and a link to the source. Running it against Claude Code's own changelog is what caught the /skill-doctor addition for this post. Schedule it, and add /skill-doctor output to the same report so usage and freshness arrive together.

# crontab: every other Monday, 07:00
0 7 */14 * 1 cd ~/dev/skills && claude -p "$(cat jobs/drift-check.prompt)" >> logs/drift.log 2>&1

One measurement closes the loop. After the first prune, read the /cost cache line across ten consecutive turns. If the skill listing is stable, the cache should report warm on every turn after the first. If it still misses, something else in the system prompt is churning — usually an MCP server re-announcing its tools on reconnect — and /skill-doctor was never going to catch that. Fix the churn first, then prune again, because a warm cache makes every remaining skill cheaper to carry.

The skill and rules files we run in production, including the evals layout above, ship in the Claude Code Production Pack; the CLAUDE.md Production Rules set is the companion for the always-loaded layer. If you want the architecture argument for what belongs in a skill versus a rule versus a subagent, our three-layer harness pattern post is the long version. Run /skill-doctor on your own setup today and delete the first thing it flags — then measure the next session with the token counter. Every product mentioned is available at wowhow.cloud — pay once, ship forever.