For most of the past four years, the story of artificial intelligence was told through training. Who had the biggest cluster, the largest model, the most expensive run. When OpenAI launched GPT-6 Astra in September, much of the coverage focused on the more than 100,000 GPUs used to train it at the Stargate site in Texas. Training is dramatic, expensive and easy to measure.
It is also, increasingly, the smaller part of the bill. Every time a customer asks a question, an agent completes a task or a developer’s code assistant suggests a line, a model has to run. That work is called inference, and it happens billions of times a day. By most industry estimates, inference now accounts for the majority of spending on AI computing. The shift is quietly rewriting the economics of the whole industry, from chip design to cloud contracts to the price an enterprise pays for each answer.
From One-Off Cost to Running Cost
The difference between training and inference is the difference between building a factory and running it. Training is a large, lumpy capital expense. A lab spends heavily for months, and the result is a model that can be used for years. Inference is an operating cost that scales with usage. The more successful a model is, the more it costs to run.
That changes who carries the risk. A training run that fails is a painful write-off. An inference business that grows faster than its margins is a much deeper problem, because every new customer adds cost. This is why investors have become so interested in the gross margins of AI providers, and why the financing strains discussed during October’s chip selloff, from Meta’s negative free cash flow to questions about the working capital of neocloud operators, matter so much.
| Training | Inference | |
|---|---|---|
| What it is | Teaching a model from data | Running the model to answer requests |
| Cost pattern | Large, periodic capital outlay | Continuous cost that grows with usage |
| Hardware priority | Raw compute and fast links between thousands of chips | Cost per token, memory bandwidth, latency, energy use |
| Where it runs | A few giant clusters | Many data centres, near users and on devices |
| Who pays | Model developers and their investors | Every customer, every query |
| Strategic question | Who can build the best model? | Who can serve it profitably at scale? |
The Price of Intelligence Keeps Falling
The single most striking number in AI economics is how fast the price of a given level of capability has fallen. Stanford’s AI Index found that the cost of running a model performing at roughly the level of the original GPT-3.5 fell from about $20 per million tokens in late 2022 to around seven cents by late 2024, a fall of more than 280 times in two years. Prices have kept falling since.
Several forces drive this. Each generation of chips from Nvidia and its rivals delivers far more throughput per dollar. Engineers have become much better at squeezing models, through techniques such as quantisation, distillation into smaller models and smarter routing between them. And competition is fierce. DeepSeek’s low-cost models jolted the market in early 2025 by showing that capable systems could be built and served far more cheaply than American labs had suggested.
Cheaper Tokens, Bigger Bills
If prices are falling so fast, why are so many companies seeing their AI bills rise? The answer is an old economic idea known as the Jevons paradox. When something becomes cheaper to use, people use much more of it, and total spending can rise even as unit costs collapse.
Two changes have supercharged that effect. Reasoning models think before they answer, generating long internal chains of reasoning that can use many times more tokens than a simple reply. Agents go further still. A system that browses, writes code, checks its own work and runs for an hour on a single task can consume more computing in one job than a chatbot used in a month. As we noted in our analysis of agentic AI in the enterprise, the business value of agents is real, but so is the meter running in the background.
For finance directors, the practical lesson is that a falling price per token tells you very little about your future AI budget. The number that matters is cost per completed task, and that depends on how many tokens your workflows consume, which models they use and how well they are designed.
The Custom Silicon Bet
Inference has also reopened the chip market. Nvidia’s GPUs remain the default for training, and its software ecosystem gives it a formidable lead. But inference rewards different trade-offs: lower cost per token, efficient use of memory and lower power draw. Those are areas where chips designed for a single company’s workloads can compete.
Every hyperscaler is now betting on this. Google’s TPUs, including the Ironwood generation it describes as built for the age of inference, run much of its own AI and are offered to cloud customers. Amazon’s Trainium and Inferentia chips power large deployments, including the Project Rainier cluster built for Anthropic. Meta has its MTIA chips and Microsoft its Maia line. OpenAI has agreed to develop custom accelerators with Broadcom. Each company is trying to cut its dependence on a single supplier and capture the margin that currently flows to Nvidia.
As we explored in our article on chiplet architectures, the move away from a GPU monoculture is technical as well as commercial. Custom chips need years of investment and a huge, predictable workload to justify the design cost. Only a handful of companies have that scale. For everyone else, the benefit arrives indirectly, through cheaper cloud prices and more choice.
Distribution Goes Multi-Cloud
The commercial side of AI is changing just as fast. OpenAI’s decision in October to sell its frontier models through Google Cloud, following earlier moves onto Amazon Web Services after Microsoft’s exclusivity ended, means the choice of cloud no longer dictates the choice of model. A company committed to Google can now run OpenAI, Gemini and Anthropic models side by side on one bill.
That is good news for buyers and an inference story at heart. Once models are available everywhere, providers compete on price, speed and reliability of serving, not just on benchmark scores. The clouds compete on how cheaply they can run other people’s models on their own hardware. Margins will go to whoever runs inference most efficiently, which is precisely why Google, Amazon and Microsoft are pouring money into their own chips.
China is running a parallel experiment under very different constraints. Export controls limit access to the most advanced American chips, so developers such as DeepSeek have focused on efficiency in model design and on domestic hardware. Its reported $15 billion funding round, backed by CATL, Tencent and other industrial groups, is a bet that serving AI cheaply at enormous scale across factories, cars and consumer apps can be a winning strategy, even without the best chips.
The Winners and the Squeezed
Who gains in an inference economy? The hyperscalers with their own silicon and vast data centre footprints are well placed, because they can capture the margin on both hardware and service. Companies that build software on top of models benefit from falling prices, as long as they price their own products around tasks rather than unlimited usage.
The squeeze falls on those caught in the middle. Model developers without their own infrastructure face competitors who can undercut them on serving costs. Neocloud operators that borrowed heavily to buy GPUs face the risk that prices for renting them fall faster than their debts. And companies that built products assuming expensive AI may find their pricing undercut by rivals using smaller, cheaper models that are good enough.
Energy runs through all of this. Inference happens all day, every day, close to users. A large share of the electricity demand that is now straining grids, a constraint we examined in our piece on the power wall facing AI, will come from running models rather than training them. The most efficient inference is also the most energy-efficient, which makes efficiency a strategic and environmental priority at once.
Inference Moves to the Edge
Not all inference will happen in giant data centres. Phones, laptops, cars and factory equipment increasingly carry chips designed to run smaller models locally. Apple, Qualcomm, Intel and AMD all now market processors around their AI capabilities, and carmakers are building in-vehicle assistants that work without a connection. For many everyday tasks, such as summarising a document, transcribing a meeting or spotting a defect on a production line, a compact model running on the device is fast, private and costs the provider nothing in cloud computing.
This creates a split market. Hard, open-ended problems will keep flowing to frontier models in the cloud, while routine work migrates to the edge, a shift that connects directly to the edge computing trends we covered earlier. Software providers will need to decide which tasks belong where, and price their products accordingly. Subscription plans built on unlimited cloud usage are already being replaced by tiers and usage caps, a sign that the cost of serving heavy users has caught up with the industry.
What Enterprise Buyers Should Do
The shift to inference gives buyers more bargaining power than they have ever had, and the companies that use it well can cut AI costs sharply without losing capability. The first step is to measure the right thing. Track cost per task, per customer interaction or per document processed, not just the headline price per token.
Second, match the model to the job. Many workloads do not need a frontier model. Routing routine tasks to smaller, cheaper models and reserving the most capable ones for hard problems can reduce costs dramatically. Third, negotiate. With models available across clouds, buyers can ask for committed-use discounts, price protection and the right to switch providers. Using existing cloud commitments to pay for AI, now possible on several platforms, can make budgets go further.
Fourth, design for efficiency. Prompt caching, batching, limits on agent run time and careful engineering of workflows can cut token use without hurting results. Finally, keep governance in view. Running several models across several clouds adds complexity in security, data protection and audit, and that work belongs in the budget too.
The Next Phase of the AI Race
The first chapter of the AI boom was about who could build the most capable model. The next will be decided by who can deliver that capability to billions of users and millions of businesses at a price that makes economic sense for both sides.
That is a less glamorous contest, fought over chip efficiency, data centre utilisation, energy contracts and cloud pricing. It is also the one that will determine whether today’s enormous investment produces profits or write-downs. For enterprises, the message is encouraging. Intelligence is getting cheaper every quarter. The companies that win will be the ones that learn to buy it wisely.
