Skill check

Check whether a skill earns its place: run the same task with and without it and compare. Skills cost context and time on every run, so a skill that doesn't change the result is worth removing.

Pick the skill and a task it should help

beswarm skill list   # note the skill's id
beswarm prompt list  # note the task prompt's id

Choose a task the skill claims to improve, such as a security review for a security-audit skill.

Run with and without

beswarm worktree add --name with-skill --workspace api
beswarm worktree add --name without-skill --workspace api
 
beswarm agent start --name with --agent claude <terminal-id> --group skilleval \
  --skill-id 2 --prompt-id 3 --wait --timeout 2400
beswarm agent start --name without --agent claude <terminal-id> --group skilleval \
  --prompt-id 3 --wait --timeout 2400

With --skill-id, the agent reads and follows the skill first; that turn finishes before the prompt is sent.

note

Check that Autoload isn't adding the skill to the "without" run. agent start sends autoload items only with --autoload, so leave that flag off both runs.

Compare

beswarm agent start --name judge --agent codex --group skilleval \
  --prompt "Compare the work in .worktrees/with-skill and .worktrees/without-skill on the same task. What did the skill change? Was it better, worse or the same? Be specific. Write to reports/skill-eval.md." \
  --wait --timeout 1200

Also compare how long each run took. If the skill only adds time, remove it, or fix it so it changes the outcome.

Let the agent choose

To test whether agents pick the right skill on their own, add a third run with --auto-skills. The agent is told to find a suitable skill in beswarm skill list:

beswarm agent start --name auto --agent claude <terminal-id> --group skilleval \
  --auto-skills --prompt "Review src/auth for security issues." --wait