This piece first appeared in German. The English version is a rewrite rather than a line-by-line translation, and the German original stays online in the archive: Inference Economics: Warum günstigere Token nicht ausreichen.

Performance equivalent to GPT-4 cost around 20 US dollars per million tokens at the end of 2022. By the end of 2025 the same capability ran at roughly 40 cents. Anyone running AI at scale in a bank or an asset manager has watched that curve fall for three years and watched the invoice go up anyway.

Cheaper per unit, more expensive in total. The mechanism behind that is not mysterious, and understanding it is what separates an architecture decision that scales from one that hits a wall in eighteen months.

Inference is the part that runs every day

Training is what happens once. A model learns patterns from an enormous body of data, at considerable expense, and then that expenditure is behind you. Inference is the counterpart: the moment a finished model is actually used. Every query, every analysis, every generated summary is one inference operation.

For a company this is the operationally relevant half. Training is capital expenditure incurred once, while inference is running cost that accrues with every use, measured in compute time, memory bandwidth and energy. With large language models that running cost is usually billed per token, meaning per unit of processed text.

How much is a token? Roughly 0.75 English words. A transcript of 10,000 words therefore comes to roughly 13,000 tokens

The question for an asset manager or a head of compliance is not what the model cost to build. It is what the model costs to operate daily at volume, and that is where the structural problem sits.

The falling unit price is a trap that only shows up at scale

Gartner expects inference on a model with a trillion parameters to cost providers over 90 percent less in 2030 than in 2025, driven by specialized inference chips, more efficient model architectures and higher hardware utilization. The forecast is plausible in its hardware logic.

The problem was never the price of the unit. It is how many units get consumed.

What are reasoning models? Older language models generate an answer directly. Reasoning models first produce a long internal chain of intermediate steps and only then arrive at the result. Quality on hard tasks improves, and token consumption per query rises sharply.

This is where the paradox forms. A reasoning model spends thousands of tokens on internal steps before it produces anything a user sees. The comparison that fits is an efficient engine installed in a much larger vehicle: less fuel per kilometer, many times the distance driven.

For enterprise deployments the consequence is concrete. Lower token prices have not reduced total cost, because reasoning models consume exponentially more tokens per task. Budget an AI stack from unit costs and the real inference bill will be underestimated systematically.

Take an earnings transcript workflow that ran on a simple completion model in 2023 and runs through a reasoning model in 2026, at ten times the token output per document. Halve the unit price over that period and the total bill is still five times what it was.

Gartner's own forecast carries the limit

The same analysis makes a second point that tends to get quoted less. A fall in commodity token prices is not the same thing as cheaper frontier reasoning. Costs for complex, compute-intensive tasks stay high, and will stay high for the time being.

That is the forecast's decisive qualification rather than a contradiction of it. Standard tasks get cheaper. The share of complex tasks grows, because users and product teams gravitate to the most capable model available even where the task does not require it. Cheaper tokens on one side, more tokens consumed on the other, and a larger invoice at the bottom.

Three levers that actually move the number

The answer is an architecture decision taken early, and it is not a budget cut.

Route by task complexity. Structured data extraction, sentiment classification and compliance keyword matching run at comparable quality on smaller and cheaper models. Sending routine, high-frequency work to smaller models and reserving the strongest model for complex reasoning returns more than a uniform frontier stack, which is also what Gartner advises.

Compress prompts and manage context. Token volume comes from the input side as much as from model output, especially in multi-stage pipelines. Redundant instructions, oversized retrieval chunks and uncompressed conversation histories all drive volume without improving quality.

Cache the repetitive structures. System instructions, standard regulatory text and fixed prompt frames repeat thousands of times across financial workflows. Prompt caching, supported by several providers, cuts the billable token count for identical input segments and changes nothing about the output.

Agentic systems are the next multiplier

Token optimization solves the structural problem only in part. The next escalation is agentic systems, where models sequence several tasks autonomously and consume fresh tokens at every step. Gartner puts agentic systems at 5 to 30 times the tokens per task of a standard chatbot. Cover architectural inefficiency with cheap commodity tokens today and agentic scaling will find the wall for you.

Regulated institutions carry an additional dimension. Anyone tied to on-premise deployment for data protection or compliance reasons takes no part in the cloud price war at all. Those institutions bear the full hardware operating cost however cheap external inference becomes, which gives them the strongest incentive of anyone to build token efficiency into the architecture.

Volume is the problem

For as long as users gravitate to the strongest model and agentic workflows multiply consumption, total costs will rise however far the unit price falls. Token optimization is a precondition for any AI architecture meant to scale, and treating it as a cost-cutting exercise gets the order of operations backwards.

Sources

Grouped by the section they support.

Opening

  • Crosley (2026), Inference Unit Economics: The True Cost Per Million Tokens, Introl, 9 February 2026, introl.com. Supports the fall from 20 US dollars per million tokens in late 2022 to 40 cents. The article gives no underlying data source for the comparison.

Inference is the part that runs every day

  • OpenAI Help Center, What are tokens and how to count them?, help.openai.com. Supports one token as roughly three quarters of an English word. The distinction between training and inference as one-off and running cost is standard and needs no source.

The falling unit price is a trap that only shows up at scale

  • Gartner (2026), Gartner Predicts That by 2030, Performing Inference on an LLM With 1 Trillion Parameters Will Cost GenAI Providers Over 90% Less Than in 2025, press release, 25 March 2026, gartner.com. Supports the forecast of more than 90 percent and its drivers, among them inference-specialized chips, model design and higher chip utilization.
  • OpenAI, Reasoning models, API documentation, developers.openai.com. Supports reasoning models generating internal tokens before the answer, which are not visible and are billed as output tokens.
  • Spark (2025), The LLM Cost Paradox: How "Cheaper" AI Models Are Breaking Budgets, ikangai, 21 August 2025, ikangai.com. Supports thousands of internal reasoning tokens per answer, the comparison with fuel-efficient engines in much larger vehicles, and falling unit prices alongside rising operating costs. The earnings-transcript calculation is an illustration.

Gartner's own forecast carries the limit

  • Gartner (2026). Supports the warning not to confuse the deflation of commodity tokens with the democratization of frontier reasoning. The tendency of users to reach for the most capable model is the author's own observation.

Three levers that actually move the number

  • Gartner (2026). Supports routing routine, high-frequency tasks to smaller and domain-specific models and reserving frontier models for complex reasoning.
  • OpenAI, Prompt caching, API documentation, developers.openai.com. Anthropic, Prompt caching, Claude documentation, platform.claude.com. Both support reusing an identical prompt prefix at a reduced input price. Prompt compression and context management are the author's own recommendation.

Agentic systems are the next multiplier

  • Gartner (2026). Supports agentic systems requiring between 5 and 30 times more tokens per task than a standard generative AI chatbot. The consequences for on-premise deployments are the author's own analysis.

Volume is the problem

  • No external source. The conclusion is the author's own.
The link has been copied!