The Skill Eval Loop

Stop guessing whether your skill helps. Measure it against test cases, then reload the edit live and re-test — without burning your session.

Why this, for you: you wrote a skill, it feels better, and you ship on vibes. Two weeks later it's quietly making outputs worse on half your real prompts and never firing on the other half. A skill fails on two independent axes — output quality and trigger precision — and eyeballing one transcript tells you nothing about either. This lesson gives you a measured loop instead.

The skill-creator framework brings eval-driven development to skill authoring with no code: define cases, benchmark pass rates, A/B-compare versions, and tune the trigger. Then Claude Code's /reload-skills collapses the inner loop to edit → reload → test inside one session.

1 Two failure surfaces, not one

Before you measure anything, name what can break. A skill can fail independently on:

Output quality — when it does fire, does it produce good results?
Trigger precision — does it activate at the right time, and stay quiet on near-misses?

Output-quality evals only matter if the skill triggers. A perfect skill that never fires scores zero in production. Eval both surfaces — they have different test sets and different fixes.

2 The eval loop: define → run → grade → iterate

Each eval in evals/evals.json has three parts: a realistic prompt with concrete details (paths, columns, context), an expected-output description, and optional input files. Start with just 2–3 cases, and add assertions after the first run — you often can't define "good" until you've seen what the skill produces.

# evals/evals.json — one realistic case, assertions added after run 1 { "skill_name": "csv-analyzer", "evals": [ { "id": 1, "prompt": "Monthly sales in data/sales_2025.csv. Top 3 months by revenue, make a bar chart.", "expected_output": "Bar chart of top 3 months, labeled axes.", "files": ["evals/files/sales_2025.csv"] } ] }

skill-creator then spawns independent agents per eval — one with the skill, one without (the baseline, or the prior version) — each in an isolated context so runs can't bleed into each other. Assertions should be specific and observable ("the bar chart has labeled axes"), never vague ("the output is good"). Grade with code-based checks for deterministic properties, LLM-as-judge for nuance, or human review as the gold standard.

The benchmark delta — pass rate, time, and tokens — quantifies cost against benefit. A 13-second overhead for a +50-point pass-rate gain is a very different trade than doubling tokens for +2 points. The delta is how you decide whether a skill earns its keep.

Then read the per-assertion pattern to know what to fix: an assertion that passes in both configs isn't discriminating (drop it); one that fails in both is a broken assertion or an impossible task; passes with, fails without is where the skill adds real value; and high variance across runs means ambiguous instructions. Generalize fixes in SKILL.md, rerun in iteration-N+1/, and compare.

Sequential judging anchors — compare blind

Grading version B right after version A introduces anchoring bias — B is judged relative to A. Comparator agents fix this: the grader sees the A and B outputs unlabeled and scores each criterion blind. Same trick works for skill-vs-no-skill, version-vs-version, or one skill across models.

3 Optimize the trigger, then reload it live

Quality is half the job; the other half is firing at the right moment. Run the description-optimization loop: generate ~20 trigger queries — 8–10 that should fire (varied phrasings: casual, formal, implicit) and 8–10 near-misses that should not (shared keywords, different intent). skill-creator scores the current description against them and suggests edits that cut both false positives and false negatives. Queries must be concrete: "format this data" is too vague; "my boss sent Q4_sales_final_v2.xlsx and wants a profit-margin column — revenue is in C, costs in D" is strong.

Now the inner-loop accelerant. Historically Claude Code scanned skill directories once, at launch, so picking up an edit meant restarting — which discards the transcript, loaded files, the task list, and all accumulated reasoning. As of v2.1.152, /reload-skills re-scans the directories in the live session, swapping in the current on-disk SKILL.md set while leaving context intact.

A restart loses	A reload keeps
Conversation transcript	Preserved
Files read into context	Preserved
Active todo / task list	Preserved
The skill set itself	Refreshed — added, edited, removed

A reload decouples the cheap, re-runnable directory scan from the expensive, lost-on-restart context. That's what collapses the authoring loop to edit → reload → test in one session — and why a SessionStart hook returning reloadSkills: true exists for skills a hook installs programmatically before the first turn.

Reload doesn't rebuild the slash-command index

A reload updates the model-facing skill set but not the / command parser. A newly added skill can be invocable by the model yet still return "Unknown skill" on direct /name invocation until a full restart. When you need it reachable by name from the command line in the same session, restart, don't reload. Also restart when debugging a mis-fire — stale prior-turn context can mask whether your edit actually changed behavior.

↪ Your win: a measured skill, tightened live

Eval both surfaces — output quality and trigger precision; a skill that never fires scores zero.
2–3 cases, assertions after run 1 — define "good" once you've seen the output, not before.
Run with-skill vs baseline in parallel, isolated contexts; read the delta (pass rate, time, tokens) to justify the skill.
Compare blind with comparator agents to kill anchoring bias when iterating versions.
Edit → /reload-skills → test in one session — but restart when you need name-level slash invocation or to debug a mis-fire.

Retrieval practice — recall, don't peek

Question 1A skill can fail on two independent axes. They are…

Question 2The eval loop says you should add assertions…

Question 3Comparator agents grade A and B unlabeled in order to remove…

Question 4/reload-skills refreshes the skill set in place but does not rebuild the slash-command parser, so a new skill may…

Question 5 · spaced recall from an earlier lessonTo give a whole team the same guardrails by default, you ship config as…

Ask me anything. Want a starter evals.json for a real skill of yours, or help deciding when the eval harness overhead isn't worth it on a single-use skill? Next in Part 3: Building Plugins — scaffolding a local plugin, wiring its extension points, and adding background monitors.