Cost per task is the new agentic AI benchmark.

The most useful thing in Artificial Analysis' v4.1 update is not a single leaderboard change. It is a change in the unit of analysis. A token is not a task. A benchmark score is not a workflow. A fast first token is not a finished deliverable. Once models are asked to run tools, browse state, write files, recover from mistakes, and keep going for dozens or hundreds of turns, the old comparison format starts to leak.

Cost per task patches that leak. It takes the cost of running the Intelligence Index and normalizes it by the number of tasks across the evaluations. That immediately makes the model market look less like a neat ranking and more like a routing problem. You can buy more intelligence, but you do not always want to buy the most expensive path to task completion.

What v4.1 is really saying

Artificial Analysis v4.1 moves the Intelligence Index toward agentic work. The current index includes GDPval-AA v2, τ³-Bench Banking, Terminal-Bench v2.1, SciCode, Humanity's Last Exam, GPQA Diamond, CritPt, AA-Omniscience, and AA-LCR. That mix matters. It gives more weight to models that can operate in messy work environments rather than only solve isolated questions.

The upgrade also changes how cost should be discussed. Provider price sheets still matter, but they are the wrong first abstraction for agents. Agents spend money through repeated context, tool traces, retries, reasoning tokens, answer tokens, cache writes, and cache hits. A price per million tokens can look cheap while the full task remains expensive. A model can also look expensive per token while being worth it if it finishes in fewer attempts.

The live leaderboard already tells a different story

The top of the agentic curve is still dominated by frontier proprietary models. That is expected. The interesting part is the shape underneath: open-weight and lower-cost models have become credible enough that the right answer often depends on failure cost, volume, and latency instead of raw rank alone.

Read the table horizontally. Claude Opus 4.8 (Adaptive Reasoning, Max Effort) sits near the top of the capability stack at $1.78 per task. DeepSeek V4 Pro (Reasoning, Max Effort) is much cheaper at $0.056 while still posting a 67.2 Agentic Index.That gap is not a trivia point. It is the difference between an agent that can be turned loose on every support ticket and a model that should be reserved for escalations.

The practical decision tree

The wrong lesson is "use the cheapest capable model." Cheap failure is still failure. The better lesson is that agentic routing needs tiers. A good system should know when to pay for maximum capability, when to route to a value model, and when to stop using a language model at all because the workflow should be a deterministic API call.

Why cached input tokens matter more for agents than chat

Prompt caching can be a small optimization in single-turn chat. In agents, it can be structural. Agents tend to carry the same policy, tool schemas, customer context, repo map, memory summary, or scratchpad across repeated calls. If the provider gives a real cached-input discount, the same architecture can become materially cheaper without changing model quality.

This is also why "one blended price" can hide a lot. Two models with similar visible output prices may behave very differently once cache reads, cache writes, reasoning tokens, and answer tokens are counted separately. v4.1's cache-aware reporting is a nudge toward the way production systems are actually billed.

The benchmark shift is healthy

Removing saturated evaluations is not a cosmetic choice. When a benchmark stops separating frontier systems, it starts rewarding the wrong thing: confidence in stale signal. Replacing older tasks with Terminal-Bench 2.1, τ³-Bench Banking, and GDPval-AA v2 makes the index harder to game and closer to the work people are now asking models to do.

GDPval-AA v2 is especially important because it is not just "answer this question." It asks for work products. Artificial Analysis describes GDPval-AA v2 as an agentic evaluation for economically valuable tasks, with the Elo scale anchored to human expert performance at 1000 and a higher turn limit for longer trajectories. That is a different world from a multiple-choice benchmark.

What to do next

If you are building an AI agent, stop choosing models from a single leaderboard column. Start with the Agentic Index. Filter by task cost. Check context and latency. Then run a small evaluation on your own workflows with the same escalation policy you plan to ship. The right model is not the one that wins the most screenshots. It is the one that lets your system finish more work per dollar without making the expensive mistakes.

FAQ

What is cost per task?

It is the average cost to complete one Artificial Analysis Intelligence Index task. It is useful because it converts model selection from abstract token pricing into a unit closer to what builders actually buy: completed work.

Does cost per task replace benchmarks?

No. It sits next to benchmarks. Capability tells you whether a model is likely to solve the task. Cost per task tells you what that capability costs when exercised in benchmark-like workloads.

Why not always use open weights?

Open-weight models are becoming excellent value choices, especially for controlled or high-volume workloads. Frontier proprietary models still matter when the failure cost is high, when the task is novel, or when the last few points of reliability are worth more than the cost gap.

Where does WhatLLM get this data?

WhatLLM uses the Artificial Analysis data APIs, including the v2 language models endpoint for Intelligence Index v4.1 headline metrics and cost-per-task data. The local snapshot is refreshed by the `/api/cron/warm-aa` path and reused across rankings, compare pages, and tools.