Inference-Time Scaling and Reasoning Models
Reasoning models are AI systems that work through a problem step by step before answering, instead of replying with the first thing that comes to mind.
What's happening now
By mid-2026, paying a model to think longer at the moment you ask it a question, rather than only building a bigger model, has become a central story of the AI industry. At its GTC conference in March 2026, NVIDIA reframed data centers as token factories and argued that reasoning models, which spend extra compute thinking before they answer, now drive compute demand far more than training larger base models, citing growth in some inference workloads of up to about 10,000 times over two years. OpenAI turned the idea into a literal product control on March 5, 2026 with GPT-5.4 Thinking and Pro, which expose five dialable reasoning-effort levels so users and developers can spend more or less thinking compute per request and trade cost against quality. The research is catching up to the slogans: a wave of 2026 scaling-law papers, including Test-Time Scaling Makes Overtraining Compute-Optimal, jointly optimizes model size, training data, and thinking budget so labs can decide when paying for longer thinking beats paying for a bigger model. Analysts at Deloitte and McKinsey describe a broad shift of AI compute from training toward inference, the practical sign that this technique now shapes hardware, products, and budgets across the field. The honest caveat is cost: thinking longer is not free, and on easy questions the extra compute buys little, so the open question is how to spend a thinking budget only where it actually helps.
What it is
Inference-time scaling is the idea that you can make these models smarter simply by letting them spend more time and computing power thinking at the moment you ask a question, rather than only making the underlying model bigger during its initial build. In short: a smaller model that is allowed to think longer can often beat a much larger model that answers instantly.
The axis is whether inference-time compute is a durable primary path to capability or a conditional amplifier subordinate to base-model quality.
The second scaling law: inference compute is a structural, lasting axisInference-time compute is a permanent, co-equal scaling axis that unlocks reasoning capabilities no amount of additional pre-training can match at equivalent cost.
This camp argues that intelligence on hard problems is not fixed at training time but is a function of how much search, verification, and deliberation a system performs when answering, mirroring how humans reason through novel problems. With pre-training returns diminishing as internet-scale text runs out, inference scaling opens a second axis that can be increased per query and cheapened as hardware improves, and Noam Brown has argued that one unit of test-time compute can equal 1,000 to 10,000 times more model size on reasoning tasks. The empirical record is the strongest evidence: o3 at high compute scored 87.5% on ARC-AGI-1, surpassing the human baseline for the first time, and Gemini Deep Think reached gold-medal standard at the 2025 IMO by exploring multiple solution paths in parallel at inference time.
Noam Brown (OpenAI, creator of Pluribus and Cicero); Google DeepMind (Gemini Deep Think, AlphaProof); the ICLR 2025 inference-scaling research community; Sebastian Raschka
A knob, not an engine: inference scaling is subordinate to the base modelInference scaling is a real and valuable lever, but its ceiling is set by base-model quality, so it improves reliability within what a model already knows rather than expanding what it can do.
By the data-processing inequality, sampling and chain-of-thought cannot add factual knowledge the base weights lack, so on knowledge-intensive tasks extended reasoning often amplifies hallucination instead of recovering missing facts. Resampling has a hard mathematical ceiling tied to the base model's false-positive rate, and the compute-optimal inference work shows a sharp phase transition: test-time compute substitutes for pre-training only on problems within the model's existing reach, while pre-training is strictly better outside it. The cost curve also runs the wrong way, with o3 on ARC-AGI-1 jumping from about $26 to $4,560 per task (roughly 172x the compute) for a gain from 75.7% to 87.5%, and ARC-AGI-2 then collapsing to near-zero, suggesting the gains were within-distribution rather than general reasoning; distillation of reasoning traces into smaller models further shows better pre-training progressively absorbs these gains.
Authors of 'The Limits of Inference Scaling Through Resampling' (arXiv 2411.17501); Zhao, Hooi, and Ng (arXiv 2509.06861); the ICLR 2025 compute-optimal inference team; Yann LeCun; Francois Chollet (ARC-AGI)
The evidence leans toward the conditional view: inference scaling is clearly a real and powerful lever, but its payoff is bounded by base-model quality, its cost curve is unfavorable at the frontier, and distillation steadily folds its gains back into pre-training, so it reads more as a knob amplifying what the base model already knows than as an autonomous primary path. That said, the first camp's strongest cases (gold-medal IMO performance, ARC-AGI-1 results in verifiable domains) are genuine and unreplicated by training alone, and every major lab has converged on the paradigm, so this remains an actively contested question rather than a settled one.
Shows the field moving from slogans to math, jointly optimizing model size, training data, and inference thinking so labs can decide when paying for longer thinking beats paying for a bigger model.
The industry's biggest hardware vendor reframed data centers as token factories and put hard numbers (~10,000x per-workload, ~100x usage) behind the claim that reasoning models letting AI think longer now drive demand more than training bigger models.
It makes inference-time scaling a literal product knob: users and developers can spend more or less thinking compute per request (none through xhigh), turning a research idea into an everyday cost-versus-quality control.