The best public leaderboard entry is a starting point, not a result on your codebase. A fair evaluation asks whether a complete agent setup can make changes your team would accept under conditions you can afford.
1. Choose representative, held-out tasks
Sample real bug fixes, features, tests and refactors across languages and components. Include tasks that require finding context, not only writing a function. Keep a held-out set for the final comparison so prompt or harness tuning does not train the process on the answers.
For each issue, save the starting commit, task description, environment, verification commands and acceptance criteria. Record task size and risk before running agents. Do not expose reference patches or later fixes in the workspace.
2. Freeze the setup and compare like with like
Record exact model variant, provider, agent and version, reasoning setting, available tools, permissions, context strategy, token budget, timeout and retry policy. Start each run from an independent copy of the same commit. If comparing complete products rather than models, disclose where their harnesses differ.
Sandbox runs and use test credentials and non-production data. A coding agent that can run shell commands may affect its environment; grant only the access the evaluation needs.
3. Define acceptance before running
Task correctness
Does the change address the issue and pass relevant tests?
Review and maintainability
Would a reviewer accept the patch without material rework?
Safety and regressions
Did the run avoid prohibited actions, exposed secrets and unrelated breakage?
Operational reliability
Did it complete within the agreed time, budget and tool permissions?
Tests are necessary evidence, not a complete definition of correctness. Where practical, blind reviewers to the agent identity. Keep rejected patches and explain why they failed.
4. Count the full workflow
For every run, record accepted or rejected outcome, tokens and cache use, model charges, tool and infrastructure costs, retries, elapsed time and human review or repair time. Calculate total spend divided by accepted changes, counting failures in the numerator. If nothing is accepted, report zero successes and total spend; the ratio is undefined.
See cost per accepted change for a worked example. Keep time and money separate unless you have agreed on a consistent cost for human time.
5. Interpret results with their limits
Report the task set, success counts, per-task results and uncertainty alongside averages. A handful of tasks can reveal failure modes but rarely establishes a reliable overall winner. Repeat runs where stochastic variation matters, and look for differences by task type rather than relying only on one aggregate score.
As context, METR's 2025 randomized study examined experienced open-source developers on their own repositories. Its results apply to that particular population, tools and period, illustrating why direct measurement matters; they are not a prediction for today's models or your team.
Sources and further reading
- SWE-bench Verified documents repository tasks and the distinction between full-system and controlled-harness comparisons.
- Terminal-Bench 4.0 describes why task, environment and resource changes require new runs.
- METR's field study describes a randomized evaluation of developer workflows.
Start with the benchmark guide, then use your own acceptance criteria to make the final choice.