Model bake-off

Keep the agent fixed and vary the model: the same agent, the same task and the same prompt, with only --model changing. It answers "is the bigger model worth it for this kind of work?" with your own code instead of a benchmark.

One worktree per model

beswarm worktree add --name model-a --workspace api
beswarm worktree add --name model-b --workspace api
beswarm worktree add --name model-c --workspace api

Save the task once

Put the task in a file under ~/.beswarm/prompts so every run gets identical wording, then find its id:

beswarm prompt list

Start one agent per model

--model is passed to the agent as its own --model flag. Use names your agent accepts:

beswarm agent start --name run-a --agent claude --model <model-a> <terminal-id> --group models --prompt-id 3
beswarm agent start --name run-b --agent claude --model <model-b> <terminal-id> --group models --prompt-id 3
beswarm agent start --name run-c --agent claude --model <model-c> <terminal-id> --group models --prompt-id 3

Terminal ids come from beswarm status. Without --model, the agent uses its default model.

Record more than the result

Time and effort matter as much as the diff. Note when each finishes:

for run in run-a run-b run-c; do
  beswarm agent wait "$run" --until done,blocked --timeout 3600
  echo "$run finished at $(date +%T)"
done

Count how often each one went blocked and needed you. That's the hidden cost of a cheaper model.

Judge blind

Ask a judge to rank the diffs without knowing which model wrote which:

beswarm agent start --name judge --agent codex --group models \
  --prompt "Rank the diffs in .worktrees/model-a, model-b and model-c against the task in ~/.beswarm/prompts/<file>. Judge correctness, test coverage and simplicity. Don't guess which model wrote which. Write a ranked verdict to reports/models.md." \
  --wait --timeout 1200

Use a different agent for the judge than the contestants, so it isn't grading its own style.

When you're done, beswarm agent close --group models and remove the worktrees.