Do not compare a Terminal-Bench 2.1 percentage directly with a 4.0 percentage. The tasks and execution conditions changed. A model can improve while its score falls on a harder or otherwise different version.
What changed from 2.1 to 4.0?
Terminal-Bench 2.1 revised 28 of the 89 tasks in 2.0. The benchmark team reports that most representative agent–model pairs improved after those fixes; that illustrates why a version change is not a simple model upgrade.
Terminal-Bench 4.0 changed task resources, fixed task and verifier issues, and removed tasks for reasons including saturation and exposed solutions. Artificial Analysis describes a 66-task 4.0 set in its coding-agent methodology. The benchmark authors say breaking changes to agent environments require rerunning trials.
Read a score with its context
A leaderboard entry represents a system: model variant, reasoning setting, agent scaffold, tool permissions, task version and evaluation rules. Compare results under the same setup when choosing a model; compare end-to-end systems when choosing a product. Confidence intervals, repeats, failed runs and cost matter too.
Historical 2.1 scores in the available WhatLLM data
This table provides historical 2.1 context, not a 4.0 leaderboard. Model families are deduplicated and the highest-scoring variant is retained; missing scores are excluded.
- 1. Claude Fable 5.1 (Max, Default Fallback)Anthropic91%
- 2. GPT-6 Astra (High)OpenAI90%
- 3. Qwen3.8 Max (0902)Alibaba89%
- 4. Gemini 3.8 Flash (High)Google88%
- 5. GLM-5.3-FlashZ AI84%
These rows use the available source's model-level labels; the feed does not include every run's scaffold, budget or confidence interval. Treat small score differences as directional, not as proof that one agent will win on your work.
Coding Index is not Coding Agent Index
Artificial Analysis's Intelligence Index v5 includes a coding category with FrontierCode 2.0 and Terminal-Bench 4.0. The separate Coding Agent Index uses its own benchmark set and methodology. Do not relabel a model's Coding Index as a coding-agent leaderboard score or treat index versions as one continuous scale.
Use the benchmark to design your own evaluation
- Shortlist candidates on the same benchmark version and comparable harness.
- Test each on held-out issues from your actual repositories with identical starting states, tools, limits and acceptance rules.
- Record tests, review outcomes, all retries, spending and elapsed time. Keep failed runs.
- Inspect failure modes and uncertainty before selecting or routing between agents.
The cost guide shows how to calculate the cost of an accepted change without combining unrelated benchmark figures.