GIP Index / AI Prices

Cheap Tokens, Expensive Tasks

Two AI models with the same price per token and the same benchmark score can differ eightfold in what a finished task costs. The price list is the wrong place to shop.

Claude Sonnet 5.5 and GPT-6.1 Sol charge the same list price: $2 per million input tokens, $10 per million output tokens, $0.10 per million cached reads. On Artificial Analysis’s new Intelligence Index they score 54.2 and 54.5.

Sonnet costs $10.02 per index task. Sol costs $1.27.

That is a 7.9x gap with no difference in sticker price and no real difference in score. My read: price per token is the most quoted number in AI and one of the least useful. You pay for a finished task, and that depends on how many tokens a model burns, how often it fails and how many times you have to ask.

The good news holds up too. Hold the task fixed and the cost of getting it done is falling faster than almost any price in the economy. Both facts matter, and the second one is why the first matters to anyone whose job is made of tasks.

The sticker and the bill

A token price is a rate. A task cost is rate times quantity, and quantity is where models differ most.

Artificial Analysis publishes both. Its v5 index, released October 27, runs nine evaluations and reports the weighted average cost per task at list prices. Its model data also shows the tokens behind that cost.

To run the whole index, Sonnet 5.5 at max effort read 30.6 billion input tokens and wrote 335 million output tokens. Sol at max effort read 1.9 billion and wrote 64 million.

Same rates, 16 times the reading and five times the writing. The bill follows the quantity: $7,981 to run the index on Sonnet, $1,124 on Sol.

Reading is the quiet part. Agentic tasks re-read their own growing context at every step, so for most models most of the bill is input, not output.

Gemini 3.8 Flash spent $5,764 running the index, and $5,027 of that was for writing context into its cache. Its output cost $471.

So a “cheap” model is not cheap if it reads and writes a lot. Here are three pairs from the same table, all at list prices:

• Gemini 3.8 Flash vs GPT-6.1 Sol. Flash’s blended price is $1.50 per million tokens against Sol’s $4.00. Flash costs $7.77 per task against $1.27, and scores 30.5 against 54.5. It is 2.7x cheaper per token, 6.1x more expensive per task and 24 points worse.
• Ling 3.1 Flash vs MiMo-V2.6-Pro. Blended prices of $0.45 and $0.54, scores of 29.5 and 29.9. Ling costs $3.19 per task. MiMo costs $0.26. Ling writes 225,000 output tokens per task to MiMo’s 121,000 and reads nearly three times as much.
• Qwen3.8 Max vs GPT-6 Astra. Astra’s blended price is 6.7 times Qwen’s ($20 vs $3). Astra still costs 38% less per task ($6.24 vs $10.02) and scores 56.9 to Qwen’s 33.6.

To be fair to the price list, it is not useless. Across all 26 priced models, the rank correlation between price per token and cost per task is 0.79. It is a decent first guess that can miss by ten places, and ten places is the difference between a model you can afford to run all day and one you can’t.

What a completed task costs

The Intelligence Index scores work. It does not record a pass or a fail for each task, so its cost per task is really cost per attempt. To get cost per completed task you need a pass rate.

ARC Prize publishes one. Its ARC-AGI-2 semi-private set has 120 visual reasoning puzzles, each scored pass or fail with two tries, and ARC reports the cost per task for every system.

Divide that cost by the share solved and you get the expected cost per solved puzzle. Stanford researchers call this cost-of-pass: cost per attempt divided by the probability of success.

Fifteen models appear in both ARC’s data and Artificial Analysis’s price table. On this measure, failure scrambles the price ranking even more than token use does. The rank correlation drops to 0.45.

• Grok 4.7 vs GPT-6.1 Sol. Grok charges $6 per million output tokens to Sol’s $10. Grok solves 61.4% at $2.01 per task, or $3.28 per solved puzzle. Sol solves 94.2% at $0.25, or $0.27. The cheaper token costs 12 times as much per success.
• Nemotron 3.5 Super VL vs GPT-6 Luna. Both charge $0.50 per million output tokens. Nemotron solves 1.5% of puzzles, so each success costs $2.84. Luna solves 59.3% and each success costs $0.10. Same sticker, 27 times the bill.
• GPT-6 Astra, the most expensive model on the list at $50 per million output tokens, costs $1.18 per solved puzzle. That is less than Beam 1.0 ($2.60), whose output tokens cost $1.20, and less than Claude Opus 5.5 ($2.02) at $20.

Two more knobs move the bill as much as the model choice does.

The first is effort. Artificial Analysis found Opus 5.5 scores 35 on its index at low effort for $0.98 a task and 58 at max for $10.04. More thinking bought more score there.

On ARC-AGI-2 it bought less: Opus 5.5 at high effort solved 93.3% for $0.41 a task, and at max solved 91.7% for $1.85. Paying 4.5 times as much for a lower score is a choice someone made on a settings screen.

The second is the harness, the software that feeds the model its task and its tools. ARC’s newer interactive test, ARC-AGI-3, reports total run costs.

GPT-6.1 Sol at max effort scored 52.7% for $7,632 in ARC’s standard harness. With its provider’s own adapter it scored 96.2% for $3,817. Same model, nearly double the score, half the money.

What about just trying again? The cost-of-pass math assumes retries are independent, and in practice they aren’t. Some tasks a model never solves.

In the Stanford paper, with a human expert priced at up to $20 per problem as the fallback, GPT-4o mini (15 cents per million input tokens) had an expected cost of $14.67 per solved AIME math problem. o3-mini, at $1.10, cost $2.03. The cheap model failed more, and the human had to finish the job.

The same paper tested the obvious fix of asking several times and taking the majority answer. Across all six datasets and the cheaper models they tested, majority voting with three or four votes produced zero economic gain. Three votes cost three times as much, and the extra accuracy never paid for itself.

Hold the task fixed and the price collapses

Now the upside, and it is large.

Take ARC-AGI-2 and ask a simple question each month: what is the cheapest system on ARC’s leaderboard that clears a given score? The answer falls in steps every time a new model ships.

The first system to reach 85% on ARC’s list was GPT-5.5 at extra-high effort in April, at $1.87 a task. By late September, GPT-6.1 Sol at medium effort cleared 85% for about 10 cents. That is an 18.6x drop in five months.

At the 50% line the fall is steeper: $1.90 in December 2025 (GPT-5.2) to about 4 cents in July (DeepSeek V4 Flash 0731), roughly 46x in seven and a half months.

These are fast even by AI standards. MIT FutureTech’s study of inference prices puts the typical decline in the price of a given level of benchmark performance at 5x to 10x a year. ARC-AGI-2 is a single benchmark that labs are racing on, so treat it as the fast end.

I covered the broader price collapse in Knowledge is free. Creativity isn’t. and won’t repeat those numbers here.

The same MIT team flags the trap this piece is about. They note that reasoning models “can have a lower per-token cost but a higher overall cost due to generating more tokens.” Epoch AI measured the trend behind it: responses from reasoning models were growing about 5x a year against 2.2x for other models, and reasoning models used around 8x more tokens.

The frontier keeps getting more expensive

Fixed tasks get cheap. The tasks don’t stay fixed.

Artificial Analysis made its index harder in October. Its v4.3 index in September had GPT-6 Astra at $3.26 per task and Claude Fable 5.1 at $7.63.

On v5, the same models cost $6.24 and about $13. Same models, harder tasks, roughly double the bill. The median model’s output tokens per task rose from 35,000 to 59,000.

ARC-AGI-3 shows where this goes. A full run of its interactive games cost $20,657 for Claude Opus 5 at a 30.2% score and $26,098 for GPT-6 Astra at 62.7%, in ARC’s standard harness. The cost of doing the newest, hardest thing is rising even as the cost of doing last year’s hardest thing falls through the floor.

One caution on every number above: these are list prices. They tell you what a buyer pays, not what the chips and power cost the provider.

The MIT team checked closed-model prices against open-weights models, where competition sets the price, and found closed prices falling faster, “hinting at non-technical competitive effects.” I found no public data that settles whether any lab sells below cost, so I’m not claiming it.

Let’s talk about the scary parts

Here is the uncomfortable comparison. ARC Prize paid a human panel the equivalent of $17 per task on ARC-AGI-2, built from a $115 to $150 show-up fee plus $5 per solved puzzle.

ARC itself says the true floor for human cost is likely $2 to $5 a task. People in its study took a median 2.2 minutes per solved puzzle.

The cheapest model clearing 85% now costs about 10 cents. Even against ARC’s $2 floor, that is a twentieth.

So what happens if that’s your job? The work at risk first is high-volume, checkable work, where a grader can say “solved” or “not solved.”

The Stanford team ran a customer service test (Tau-bench) and priced the human at $2.06 per ticket: $20.59 an hour from the Bureau of Labor Statistics times six minutes.

With models plus a human for the failures, the best expected cost per resolved ticket fell to $0.73 by early 2025. That is a third of the human cost, not a hundredth, because someone still had to handle the tickets the models fumbled.

That fallback is the relevant detail for workers. In the same paper, human experts stayed fully essential on four of six benchmarks, because at least one problem in each was something no model solved. The machine takes the easy share, and a smaller payroll keeps the hard part.

On longer agentic work the picture is mixed. METR, pricing a human at $143.61 an hour, found more than 80% of successful agent runs cost under a tenth of the human’s pay.

Toby Ord ran the numbers the other way and found o3 cost about $350 an hour at the edge of what it could do, against about $120 an hour for a human software engineer.

As he put it, that’s “a lot of money to pay for an agent that fails at the task you’ve just paid for 50% of the time.” At the edge of capability, the failures are the bill.

Where does creativity price in? Cost-of-pass needs a definition of “pass.” The Stanford authors say so themselves: their framework assumes success is binary.

Work where nobody can write the grader, where quality is taste or judgment, doesn’t fit the formula. That’s where people still set the price, at least until someone writes the grader.

What this means for the GIP

The Gross Intelligence Product multiplies AI electricity by compute per watt by capability per unit of compute. The last term is a quality term, and this piece is why we will measure it per task, not per token. A model that reads 16 times as many tokens for the same score is doing far more computing for the same result, and the quality term should say so.

The model-by-date cost and score data behind this article will seed that quality series. We are not publishing a reading yet. The method is on the methodology page, and the first reading will say exactly which models and tasks it uses.

What to do with this

If you are buying AI for real work, stop comparing price sheets. Run 50 to 100 of your own tasks through two or three models, count every token including cached reads, decide in advance what counts as done and divide total cost by tasks done. That number is the only price that matters, and it often reverses the price sheet.

Then test effort levels and harnesses before you test new models. Max effort is not always the best value, and the vendor’s own adapter can halve the bill.

Skip blanket retries. Budget for a human on the failures, because that cost is real and it is where the savings get eaten.

Pay for the finished task, not the token. That’s my recommendation.

Sources

1. “Announcing Artificial Analysis Intelligence Index v5,” Artificial Analysis, Oct 27, 2026. https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v5
2. “Announcing the Artificial Analysis Intelligence Index v4.3,” Artificial Analysis, Sep 7, 2026. https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3
3. “AI Models: Intelligence, Price and Cost to Run the Intelligence Index” (v5 data), Artificial Analysis, read Nov 1, 2026. https://artificialanalysis.ai/models
4. “ARC-AGI Leaderboard” and evaluations data, ARC Prize, read Nov 1, 2026. https://arcprize.org/leaderboard and https://arcprize.org/media/data/evaluations.json
5. “Announcing ARC-AGI-2 and ARC Prize 2025,” ARC Prize (Greg Kamradt), Mar 24, 2025. https://arcprize.org/blog/announcing-arc-agi-2-and-arc-prize-2025
6. “ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems,” Chollet, Knoop, Kamradt, Landers, Pinkard, May 2025. https://arxiv.org/abs/2505.11831
7. “Cost-of-Pass: An Economic Framework for Evaluating Language Models,” Erol, El, Suzgun, Yuksekgonul, Zou (Stanford), 2025. https://arxiv.org/abs/2504.13359
8. “The Price of Progress: Algorithmic Efficiency and the Falling Cost of AI Inference,” Gundlach, Lynch, Mertens, Thompson (MIT FutureTech), 2025. https://arxiv.org/abs/2511.23455
9. “LLM responses to benchmark questions are getting longer over time,” Epoch AI, Apr 17, 2025. https://epoch.ai/data-insights/output-length
10. “Hourly Costs for AI Agents,” Toby Ord, Dec 22, 2025. https://www.tobyord.com/writing/hourly-costs-for-ai-agents
11. “Measuring AI Ability to Complete Long Tasks,” Kwa et al. (METR), Mar 2025. https://arxiv.org/abs/2503.14499