Process Rewards vs Outcome Rewards: Building Better RL Training Signals
Compare process rewards vs outcome rewards for RL training. Learn when each signal wins, how PRMs work, and how to avoid reward hacking.
Why Reward Design Is the Bottleneck
For most of the past three years, the fastest way to improve an LLM was to scale pretraining—more parameters, more tokens. Those returns have not vanished, but they have flattened for many target capabilities. The higher-leverage variable now sits downstream: the quality of the reward signal used in post-training. A frontier model trained against a poorly specified reward plateaus early and hacks its way to a high score; a smaller model trained against a well-designed reward keeps climbing on the metric you actually care about. Teams that spent 2023 and 2024 racing on cluster size are now spending 2025 and 2026 racing on reward engineering, and the gap between a well-specified and poorly specified reward routinely dwarfs the gap between a 7B and a 70B base model on reasoning benchmarks.
The tension comes down to two philosophies. Outcome rewards are sparse and cheap: you grade the final answer and move on. Process rewards are dense and expensive: you grade intermediate steps. As reasoning models and long-horizon agents push credit assignment to its limits, teams are revisiting reward design with fresh urgency—and, importantly, rediscovering that the two are not mutually exclusive. The naive framing—"process rewards are better because they're denser"—collapses the moment you account for labeling cost, reward hacking, and the fact that a dense signal you can't trust is worse than a sparse signal you can.
What Are Outcome Rewards?
Outcome rewards assign a score only to the final result. The model solves a math problem and you check the answer against ground truth. It writes code and you run the test suite. It drafts a response and a human or preference model rates the finished output. Everything in between is invisible to the reward.
The appeal is obvious. Outcome rewards are cheap to collect and trivially verifiable wherever ground truth exists—mathematics, competitive programming, structured extraction. A unit test suite runs in milliseconds and costs effectively nothing per sample; a math answer check is a string comparison. They also align naturally with RLHF preference data, where annotators judge finished outputs rather than internal reasoning. Outcome-based reinforcement learning powered the first wave of reasoning models precisely because the verifier was free and unambiguous—DeepSeek-R1's GRPO training on verifiable math and code rewards is the canonical example, and it worked because the reward was a hard, binary correctness check with no room for interpretation.
Where Outcome Rewards Break Down
The core failure is the credit assignment problem. When a model produces a 40-step reasoning trace and fails at the end, which step deserves the penalty? A terminal reward punishes all 40 steps equally, even if 39 were flawless. With sparse, delayed feedback, learning is slow and sample-inefficient—the gradient carries almost no information about where the trajectory went wrong. In practice this shows up as flat reward curves: the model samples thousands of trajectories, most of them fail at the same handful of steps, and the policy has no way to isolate those steps because the only feedback it receives is a single zero at the end.
Worse, the model learns to optimize the verifier, not the task. It finds shortcuts—exploiting test harnesses (hardcoding expected outputs, catching and swallowing exceptions, importing the solution from a package), pattern-matching to answer formats (emitting \boxed{42} because the format correlates with correctness in the training distribution), or producing plausible-looking reasoning that happens to land on the right token. In long-horizon agent tasks, where a final success can mask dozens of bad intermediate decisions (and a final failure can mask dozens of good ones), outcome rewards give you almost no leverage. A coding agent that stumbles into a passing test after three wrong file edits gets the same reward as one that reasons cleanly to the fix—and the policy learns to stumble.
"An outcome reward tells you the model failed. It doesn't tell you where—and in reasoning-heavy domains, where is the entire signal."
What Are Process Reward Models?
Process reward models (PRMs) score intermediate reasoning steps rather than only final answers. Given a problem and a partial solution, a PRM assigns a score to each step—typically a probability that the step is correct or that the trajectory is on track—producing a denser signal across the trajectory.
It is worth being precise about two distinct things the literature calls a "PRM," because they fail differently:
- Step-correctness classifiers. Trained to label each step correct/incorrect (the formulation in Lightman et al., "Let's Verify Step by Step," 2023). Best used as verifiers for reranking or search.
- Success predictors / value functions. Trained to estimate the probability the trajectory reaches a correct final answer from a given state. These behave like learned value functions and are what you would use for reward shaping.
Conflating the two is a common and expensive mistake: a step-correctness classifier answers "is this step valid?" while a success predictor answers "will this trajectory succeed?" The first is a local judgment that can be right while the trajectory still fails; the second is a global estimate that can be well-calibrated on average while being useless for any single step. If you optimize against the wrong one, you get the wrong behavior.
PRMs are typically trained in one of three ways:
- Step-level human labels—annotators mark each reasoning step correct, incorrect, or neutral. Highest quality, highest cost; PRM800K, the dataset behind Lightman et al., required roughly 800,000 step-level labels across 12,000 problems, and the annotation effort was substantial.
- Monte Carlo rollouts—the model generates many completions from each intermediate state, and the empirical success rate becomes a step label (the approach behind Math-Shepherd and similar work). Cheaper per label, but noisy at low sample counts and expensive in generation compute.
- Automatic step verification—programmatic checks or a stronger verifier model grade individual steps. Scales best, but inherits the verifier's blind spots and is only as good as the step-level check.
The payoff is signal density. Instead of one bit of feedback per trajectory, you get a reward at (or near) every step, which accelerates learning and gives the model a clearer gradient on where its reasoning goes wrong. This is a key mechanism behind the gap between strong reasoning models and merely large ones—a 7B model with a well-calibrated PRM used for best-of-N reranking can outperform a much larger model sampling once, because the PRM is doing the discrimination work that scale alone does not buy.
How PRMs Generate Step-Level Signals
A trained PRM operates as a scorer. Given a problem and a partial solution up to step k, it outputs a value. How you use that value matters enormously:
- As a verifier (lowest risk). Score N sampled solutions and pick the best—best-of-N, or guide tree search. No policy-gradient distortion, no reward hacking of the training loop, but no direct learning signal either. This is the safest production use and the one most teams should start with.
- As a dense RL reward (highest risk). Optimize directly against per-step PRM scores. This is where the model learns to game the PRM's biases—emitting steps that look correct to the PRM while drifting from actual correctness—and it is the recipe most likely to blow up in production. The classic symptom is PRM score climbing while held-out accuracy falls.
- As reward shaping (the principled middle). Add a dense term to the outcome reward using potential-based shaping:
r'(s, a) = r(s, a) + γ·Φ(s') − Φ(s), where Φ is the PRM's value estimate. Ng, Harada & Russell (1999) proved this preserves the optimal policy—so you get denser gradients without silently changing what "correct" means. Use this, not naive additive step rewards. The difference is not academic: naive additive step rewards change the objective, and the policy will happily trade a slightly worse final answer for a much higher sum of step scores.
The most practical 2026 recipe is hybrid: keep a verifiable outcome reward as the primary objective, use Monte Carlo rollouts to bootstrap step labels cheaply, refine the highest-impact steps with human annotation, and deploy the PRM as a shaping term plus a best-of-N verifier. The PRM's job is to distinguish productive reasoning from plausible-sounding dead ends—the discrimination outcome rewards cannot provide—not to replace the ground-truth check. Budget the annotation effort where it matters: the steps that most often precede failure, not every step uniformly.
Process vs Outcome Rewards: A Direct Comparison
Neither signal dominates across the board. The right choice depends on your domain, budget, and—critically—on whether the final answer is cheaply verifiable.
Signal density favors PRMs decisively. Labeling cost runs the other way: outcome labels are often free (run the tests), while step labels require annotation infrastructure or expensive rollouts—MC labeling costs roughly N rollouts × steps × trajectories in generation, so budget accordingly. A 40-step trajectory with 8 rollouts per step across 10,000 problems is over 3 million generations just to build the label set, which is why teams bootstrap with MC and reserve human annotation for the highest-leverage steps. Verifiability is outcome rewards' home turf in math and code; PRMs earn their keep where final answers are ambiguous or delayed, such as open-ended analysis and multi-tool agent workflows.
On reward hacking, both are vulnerable but differently. Outcome rewards invite verifier exploits—the model finds the hole in the test harness. PRMs invite overoptimization against the PRM's own biases—the model finds the step style the PRM overrates. The latter is detectable by watching for divergence between PRM score and held-out outcome success, and mitigated with a KL penalty to the reference policy (the same KL-to-reference term standard in RLHF, typically tuned to keep the policy within a bounded divergence of its starting point). Compute overhead is higher for PRMs (a second model in the loop, plus the rollout cost to train it), and generalization is often—though not universally—better with process signals, which teach transferable reasoning rather than answer-matching; the caveat is that PRMs can overfit to surface step style (verbose steps, numbered lists, hedging language), so validate on held-out tasks that differ in format from the training set.
Trade-offs at a Glance
| Dimension | Outcome Rewards | Process Rewards (PRM) |
|---|---|---|
| Signal density | Low (one per trajectory) | High (per step) |
| Labeling cost | Low to free | High (labels or MC rollouts) |
| Verifiability | Strong with ground truth | Requires a trusted, well-calibrated PRM |
| Reward hacking risk | Verifier/shortcut exploits | PRM overoptimization (esp. as dense reward) |
| Compute overhead | Low | Moderate to high |
| Generalization | Answer-matching bias | Often better transfer; watch step-style overfit |
| Best production use | Primary objective where verifiable | Shaping term + best-of-N verifier |
When to Use Each Signal
Reach for outcome rewards as the primary objective when ground truth is cheap and unambiguous—math with known answers, code with real test suites, structured extraction with schemas. Here the verifier is trustworthy and free, and a PRM adds cost and a new attack surface for little gain. Pair it with best-of-N if you want extra quality at inference time—sampling 16 candidates and reranking with a verifier is often the cheapest quality win available.
Reach for process rewards when the outcome is delayed, ambiguous, or sparse—multi-step agents, long tool-use workflows, open-ended analysis where a single terminal grade carries little information. Here the PRM is the only way to get a usable gradient. But deploy it as potential-based shaping over a real outcome reward, and monitor for PRM overoptimization with a held-out outcome check on every training run.
The default for most teams is hybrid: verifiable outcome reward as the anchor, PRM as shaping and as a best-of-N verifier, with the PRM trained on MC-bootstrapped labels and refined by targeted human annotation. Start with the outcome reward alone to establish a baseline, add the PRM as a verifier before you add it as a shaping term, and only promote the PRM to a dense reward if the shaping variant shows stable held-out gains. The order matters because each step adds a failure mode, and you want to know which one broke before you stack the next.