GPU & Compute
Inference efficiency stopped being a cost line and became a capacity strategy
When power is the binding constraint, every watt saved per token is capacity you did not have to contract for. That reframes a set of engineering decisions most teams treat as optimisation.
Set two facts from this month beside each other. NVIDIA claims up to a 10x reduction in inference token cost with Rubin. Gartner expects 40% of AI data centres to be power-constrained by 2027.
In a world with abundant power, the first is a margin improvement. In a power-constrained one, it is how much you can serve at all.
The substitution nobody has priced
Efficiency and capacity have always been substitutes in principle. What changes when power is scarce is the exchange rate. If you cannot buy another megawatt on any timeline that matters, halving the energy per request is the only remaining way to double throughput.
That makes a set of familiar engineering choices strategic rather than incremental: quantisation, distillation to a smaller task-specific model, caching and deduplicating requests, routing easy queries away from frontier models, batching aggressively. All well understood. All routinely deprioritised because the payoff was a cost percentage.
Under a power ceiling, efficiency work is not cost reduction. It is the only capacity you can build without an interconnection agreement.
Where the biggest wins sit
Not in the model, usually. In the system around it. Most production deployments send every request to the largest available model regardless of difficulty, re-compute answers they have already produced, and carry context far larger than the task requires. Fixing those is unglamorous and frequently yields more than a hardware generation.
Agentic systems are the extreme case, and the opportunity is correspondingly large. An agent that re-reads its entire context at every step is burning energy on tokens it has already processed. Context management is an efficiency intervention that most teams have not yet treated as one.
The uncomfortable trade
Efficiency and assurance pull against each other. The verification passes and second opinions that make autonomous systems defensible cost tokens, and tokens now cost capacity as well as money.
Teams that squeeze efficiency purely to serve more traffic will find they have spent their improvement on volume and have nothing left for the controls. The better allocation, for anything operating where failure is expensive, is to bank part of the efficiency gain against assurance rather than throughput.
Sources
Written by the Devence Lab research team.
