Prompt A/B test

Test whether a prompt change actually helps: run two versions of the same prompt on the same task with the same agent, and compare.

Write both versions as files

Keep each version under ~/.beswarm/prompts, so the wording is fixed and reusable:

~/.beswarm/prompts/refactor-v1.md
~/.beswarm/prompts/refactor-v2.md

Change one thing between them: add a checklist, require tests first, or ask for a plan before code. If you change three things, you won't know which one mattered.

beswarm prompt list   # note the ids of both versions

Run each version on its own branch

beswarm worktree add --name prompt-v1 --workspace api
beswarm worktree add --name prompt-v2 --workspace api
beswarm agent start --name v1 --agent claude <terminal-id> --group ab --prompt-id 4 --wait --timeout 2400
beswarm agent start --name v2 --agent claude <terminal-id> --group ab --prompt-id 5 --wait --timeout 2400

Run it more than once

One run per version is an anecdote. Agents vary from run to run, so repeat on two or three tasks, or a few times on the same task, before deciding. Name runs so they stay distinct: v1-a, v1-b, v2-a, v2-b.

Compare

beswarm agent start --name judge --agent codex --group ab \
  --prompt "Compare the diffs in .worktrees/prompt-v1 and .worktrees/prompt-v2 for the same task. Which is more correct, better tested and simpler? List concrete differences. Write to reports/prompt-ab.md." \
  --wait --timeout 1200

Keep the winner as the prompt file your team uses, and delete the loser so nobody runs it by accident.