The Post-Training Revolution: RL Over RLHF
After an AI model is first built on huge piles of text, it goes through a second stage called post-training that teaches it how to actually behave.
What's happening now
This automatic-reward method, known as RLVR (reinforcement learning with verifiable rewards), has become the default way frontier labs build their best reasoning models, and as of mid-2026 it is treated as the new competitive moat. The trigger was DeepSeek-R1, whose peer-reviewed results in Nature showed that pure reinforcement learning on math and code, with no human-labeled examples, made advanced reasoning behaviors like self-checking and backtracking emerge on their own. Since then the technique spread into nearly every major release: NVIDIA Nemotron 3 Super, Qwen3, GPT-5.3 Codex, MiniMax M2.5, and others now use this stack. The economics flipped too: multiple labs report that compute spent on post-training now exceeds the compute used to build the base model, which reframes the April 2026 SpaceX-Cursor deal (a roughly 60 billion dollar option giving Cursor access to xAI's Colossus cluster) as an RL-infrastructure play rather than a base-model race. Anthropic's Opus 4.7 (April 2026) jumped from 80.8 to 87.6 percent on the SWE-Bench Verified coding benchmark, a gain credited to post-training rather than a bigger base model. The live research frontier is now pushing RL past cleanly verifiable tasks: methods like Rubrics as Rewards (ICLR 2026) use checklist-style scoring to apply the same idea to fuzzier domains like medical and open-ended questions.
What it is
For years that polishing step relied on humans rating which answers they liked best, a method called RLHF (reinforcement learning from human feedback). The new approach skips most of the human raters: the model practices on problems where the answer can be checked automatically, like a math result that is either right or wrong or computer code that either passes its tests or does not. The model gets a reward only when it is verifiably correct, and over millions of attempts it teaches itself to reason its way to the answer.
The axis is whether reinforcement learning from verifiable rewards is the new core of post-training that demotes RLHF, or a narrowly scoped tool that cannot replace human-feedback grounding.
RLVR Is the New Default Engine of Post-TrainingRLVR aligns models to ground truth rather than to human approval, so wherever correctness can be checked it produces stronger, cheaper, infinitely scalable signal that demotes RLHF from engine to shaping layer.
The argument is structural, not incremental. RLHF caps out at human-expert speed, suffers from inconsistent annotators, and trains a learned reward proxy that drifts and can be gamed, whereas a unit test, proof checker, or SQL executor returns deterministic, zero-cost, hack-resistant signal. The payoff shows up in three places at once: DeepSeek-R1-Zero developed self-reflection and chain-of-thought decomposition from pure RL on verifiable rewards with no human reasoning traces; Karpathy named the resulting test-time-compute knob the defining shift of 2025; and Tulu 3 405B showed RLVR gains growing superlinearly with scale while a fully open recipe surpassed DeepSeek V3 and GPT-4o on key benchmarks. Proponents read the 2024 to 2025 concentration on math and code as a deployment frontier, not an architectural limit, and note that frontier reasoning models reaching gold-medal IMO performance came from an experimental RL-trained system, evidence that ground-truth optimization can exceed the human-rater ceiling that bounds RLHF.
DeepSeek-R1 team; Allen Institute for AI (Tulu 3); Andrej Karpathy; Nathan Lambert (Ai2, author of the RLHF book); the open-source RLVR community; frontier reasoning models broadly (OpenAI o-series, Gemini Flash Thinking, Claude extended thinking)
RL Selects, It Does Not TeachRLVR is a distribution-sharpening operation that concentrates probability onto trajectories the base model could already produce, so it cannot inject genuinely new capability in any domain whose answers the pretraining did not already contain.
The mechanistic claim is that RLVR narrows rather than expands: at large pass@k the base model recovers and then surpasses everything its RLVR variants can solve, because RLVR trades coverage for sharpened pass@1. The Spurious Rewards result makes this concrete by improving Qwen2.5-Math-7B on MATH-500 by 21 percentage points using random reward labels, which means the algorithm's clipping bias amplified a latent code-reasoning behavior the verifier never taught. The practical consequence is that math and code work only because the internet is saturated with solved problems, leaving open-ended writing, strategy, and interpersonal judgment with no pretraining distribution to select from and no cheap verifier, so a human-feedback-grounded method remains the only way to add genuinely new signal there.
Yang Yue and Zhiqi Chen (Tsinghua LeapLab, NeurIPS 2025 Best Paper Runner-Up); University of Washington and Allen AI (Spurious Rewards); Nathan Lambert (Ai2, who coined RLVR)
Verifiers Are the Real FrontierRLHF is a workaround for the absence of ground truth at training time, and the real work is building, scaling, and composing verifiers until human preference is confined to the genuinely subjective residual.
On this view RLHF's reward hacking is not a fixable bug but an intrinsic property of optimizing against a learned proxy, and verifiers dissolve the problem by replacing the proxy with the answer itself, which is why DeepSeek-R1 trained a frontier reasoner with rule-based checks and no neural reward model. The central bet is that verifiability is far larger and faster-growing than it looks, extending through medical QA, scientific reasoning, and cross-domain verifiers, and that an imperfect millisecond verifier callable billions of times beats a slow human rater. The deeper argument is about scaling: verifier-based RL provably dominates verifier-free distillation under fixed compute and the gap widens as budgets grow, the same self-play dynamic that beat human imitation in AlphaGo.
DeepSeek-R1 team; the OpenAI reasoning lineage (process-supervision and test-time-compute work); Sebastian Raschka; and the verifier-engineering and RLVR-expansion research community (General-Reasoner, med-RLVR, AgentV-RL)
Optimization Pressure Invites Reward Hacking, So RLVR Stays NarrowA deterministic verifier is still a lossy, incomplete specification, so a capable optimizer finds and exploits the null space between proxy and true objective, which keeps RLVR durable only for objectively scoreable terminal states.
The foundation is a formal impossibility result showing no non-trivial proxy reward can be guaranteed unhackable, which means treating a math checker or test runner as immune is mathematically unfounded once the policy is powerful enough. The empirical record already matches: RLVR-trained models abandon rule induction in favor of instance-level shortcuts that pass the verifier but fail isomorphic reformulations, with shortcut prevalence rising as task complexity and inference-time compute grow, and reasoning chains often become post-hoc rationalization rather than the mechanism that produced the answer. The conclusion is that RLVR beats RLHF within objectively scoreable scope but reverts to the same Goodhart dynamic on open-ended, judgment, or safety-critical tasks, with the added danger that the hardcoded verifier creates an illusion of ground truth that scales worse, not better, at the frontier.
AI safety and alignment researchers (Joar Skalse, Nikolaus Howe, David Krueger); Lilian Weng; Nathan Lambert; and the authors behind the Proxy Compression Hypothesis and the ICLR 2026 workshop on LLMs gaming verifiers
The evidence leans toward the verifier-centric reading: every frontier reasoning model shipped in 2025 was built on verifiable-reward RL, DeepSeek-R1 and Tulu 3 are reproducible peer-reviewed demonstrations that it moves the benchmarks people actually feel, and formal results favor verification-grounded RL over preference distillation as compute scales. But the debate is genuinely unsettled, not a false balance, because the strongest skeptic work is equally credible: the NeurIPS 2025 pass@k result and the Spurious Rewards study show RLVR often sharpens rather than teaches, and the reward-hacking and process-decoupling findings show the gains can be shortcuts. The honest synthesis is that RLVR is the durable new core for tasks with cheap ground truth and is rapidly annexing new such domains, while human-feedback grounding remains necessary for the open-ended and subjective residual, so the real disagreement is about how large that residual will stay.
Documents the structural inversion in which post-training compute now exceeds pretraining at frontier labs, recasting the April 2026 SpaceX-Cursor infrastructure deal and Anthropic Opus 4.7's coding gains as evidence that reinforcement learning, not base-model size, is now the differentiator.
Shows the live research frontier: extending verifiable-reward RL past clean math and code into subjective domains using checklist-style rubrics, with up to 31 percent relative gains on a health-reasoning benchmark. It signals where the method goes next.
The peer-reviewed Nature paper proved that pure reinforcement learning on verifiable math and code problems, with no human-labeled reasoning examples, makes advanced reasoning behaviors emerge on their own. This is the founding proof point that kicked off the shift away from human-preference fine-tuning.