Reinforcement Learningprocess reward modelsoutcome reward modelsPRM vs ORMLLM reasoning

Process Reward Models vs Outcome Rewards: What Actually Improves LLM Reasoning

Compare process reward models vs outcome rewards for LLM reasoning: credit assignment, reward hacking, research evidence, and how to choose for your pipeline.

The Core Question: Where Should the Reward Come From?

Reward design is the highest-leverage decision in reinforcement learning for LLM reasoning. You can tune hyperparameters, swap optimizers, and scale rollouts, but if your reward signal points at the wrong thing, the model will optimize for the wrong thing—efficiently and at scale.

The debate has crystallized around two architectures. Process reward models (PRMs) score every intermediate reasoning step. Outcome reward models (ORMs) score only the final answer. The question practitioners actually face is not which one is theoretically superior, but which one earns its cost in a given training pipeline.

The thesis here is straightforward: verifiable outcome rewards are the default recipe for training reasoning models, and PRMs are most valuable as inference-time verifiers and credit-assignment aids—not as a wholesale replacement for outcome signals. PRMs improve step-level selection and can sharpen credit assignment, but they demand far more labeling and compute, and their advantage is strongest in reranking settings and weakest when the generator itself is RL-trained. Most teams should establish a verifiable-ORM baseline before investing in PRM infrastructure. (See: How to Build an Outcome Reward Model Baseline.)

This article breaks down how each reward signal shapes reasoning behavior, what the research evidence actually supports, and how to decide which fits your use case.

Definitions: Process Reward Models vs Outcome Reward Models

Both are reward models—learned or programmatic functions that assign a scalar score to model behavior. They differ in where that score attaches.

What a Process Reward Model Scores

A process reward model assigns a reward to each intermediate reasoning step, not just the final answer. Given a chain of thought with steps $s_1, s_2, \dots, s_n$, a PRM produces a score for each step or step transition, estimating whether that step moves the solution toward a correct result.

PRMs are typically trained as supervised classifiers over step labels, and the labels come from one of two sources:

  • Human step-level annotation — labelers mark each step as correct, incorrect, or neutral (e.g., the PRM800K dataset behind Lightman et al., 2023).
  • Monte Carlo rollouts — sample many completions from each intermediate step and use the empirical success rate as a soft label (the Math-Shepherd approach, Wang et al., 2024). This produces labels automatically but costs rollout compute and inherits the base policy's own success distribution.

Note the pipeline: rollouts or humans produce labels; the PRM is then a supervised model trained on those labels. Critically, a PRM is an architecture, not a single use case. It can serve as a verifier at inference time (reranking candidate solutions by step scores) or as a dense reward signal during RL training. These are different products with different economics—reranking is a one-time inference cost, RL reward is a per-token training cost. (See: LLM Verifiers and Best-of-N Reranking.)

What an Outcome Reward Model Scores

An outcome reward model scores only the final answer. It sees the full response and returns a single signal: correct or incorrect, or a probability of correctness.

ORMs come in two flavors. For verifiable tasks—math with a checkable answer, code that passes unit tests, formal proofs—the "reward model" can be a deterministic checker requiring no learned model at all. This is the basis of RL with verifiable rewards (RLVR), the recipe behind recent frontier reasoning models. For open-ended tasks, you train a learned verifier on (response, label) pairs, which is far cheaper to label than step-level data.

The distinction matters because it determines both what the model can learn from and what it costs to build.

Side-by-side diagram comparing a PRM assigning scores to each reasoning step versus an ORM assigning
Side-by-side diagram comparing a PRM assigning scores to each reasoning step versus an ORM assigning

How Each Reward Signal Shapes LLM Reasoning

The architecture of the reward determines the shape of the learning signal—and the shape of the learning signal determines what the model learns to do.

Credit Assignment and the Sparse-Reward Problem

ORMs produce a sparse reward: one signal at the end of a potentially long reasoning chain. If a 20-step solution fails, the ORM says "wrong" without indicating which of the 20 steps caused the failure. The model must infer credit assignment from aggregate outcomes across many samples.

PRMs produce a dense reward: per-step feedback that localizes errors. If step 7 introduces a faulty assumption, the PRM can penalize step 7 directly. In policy-gradient terms, a denser, lower-variance signal per trajectory can reduce the variance of the update estimate—the core mechanical argument for process rewards. (See: Reinforcement Learning from Human Feedback (RLHF) Explained.)

The practical consequence, in principle: PRMs can yield models more robust to intermediate errors, because training explicitly discourages locally bad steps even when the final answer happens to be right. But this cuts both ways—a PRM that rewards steps which look correct in isolation can entrench a confident wrong turn, since a plausible step is not the same as a correct one.

Reward Hacking and Failure Modes

Both architectures are hackable, but differently.

ORMs are vulnerable to outcome-level gaming: the model learns to produce answers that look correct to the verifier—guessing plausible final values, exploiting answer-format shortcuts, or reverse-engineering patterns in the verifier's training distribution. With a weak learned ORM, this can dominate training. (See: Reward Hacking in LLM Training.) Verifiable rewards reduce but do not eliminate this: a deterministic checker can still be gamed through format exploits or spurious-but-passing answers.

PRMs are vulnerable to step-level gaming and label noise: the model learns to produce steps that look correct to the step scorer—verbose but plausible-sounding reasoning, stylistic mimicry of high-scoring steps, or "reward hacking the process" without genuinely improving logic. Noisy step labels compound the problem, since the PRM learns to imitate annotation artifacts, and Monte Carlo labels inherit whatever shortcuts the base policy already takes.

The failure mode shifts with the reward architecture: outcome rewards invite answer-guessing, while process rewards invite plausible-step fabrication. Neither is immune—the question is which failure mode you can detect and afford to correct.

The Evidence: What the Research Actually Shows

The empirical picture is more nuanced than either camp claims, and it depends critically on whether the reward is used for reranking or for RL training.

Math and Multi-Step Reasoning Benchmarks

The landmark result is Lightman et al., "Let's Verify Step by Step" (2023), which trained process supervision models on the PRM800K dataset and compared them against outcome supervision on the MATH benchmark. Process supervision outperformed outcome supervision for best-of-N reranking—the PRM selected correct solutions more reliably than an ORM of comparable setup. An earlier result, Uesato et al. (2022), found a similar but smaller advantage for process-based reranking over outcome-based reranking.

The crucial caveat: these are verifier/reranking results, not training results. They show a PRM is a better selector among candidate solutions. They do not show that process rewards produce a better generator under RL. Later work (e.g., Math-Shepherd, Wang et al., 2024) showed that automatically-labeled process rewards can improve both verification and training, but the gains over a strong outcome baseline narrow once the generator is itself RL-trained. When the policy already produces mostly-correct reasoning, the incremental value of dense step supervision shrinks.

The Frontier Shift Toward Verifiable Outcomes

The most important recent evidence runs in the opposite direction from the original PRM hype. The large reasoning gains in 2024–2025—DeepSeek-R1 and the o-series-style models—came primarily from RL with verifiable outcome rewards on tasks where correctness is cheaply checkable (math, code, logic). In these regimes, a deterministic outcome checker provides an essentially unhackable, infinitely scalable reward, and the model learns long chains of thought without any step-level labels at all.

The lesson is not that PRMs are wrong; it is that when the outcome is verifiable, the outcome reward is usually the better ROI, and process rewards are best reserved for (a) reranking at inference, (b) domains where the outcome is not cheaply verifiable, and (c) credit-assignment diagnostics. A PRM's advantage is largest exactly where you cannot verify the answer cheaply—the opposite of the settings that drove the frontier.

Choosing Between Them

Use this as a decision rubric:

  • Is correctness cheaply verifiable? (unit tests, exact-match math, formal proof checker) → Start with RLVR / verifiable ORM. This is the default and the highest-ROI path.
  • Do you need a better selector than generator? (best-of-N, agentic search, tree search) → A PRM as inference verifier is often worth the cost; this is its strongest documented use.
  • Is the outcome expensive or impossible to verify? (open-ended reasoning, long-horizon agents, partial-credit tasks) → Learned ORM first, then PRM if step-level credit assignment is the bottleneck and you can afford step labels.
  • Label budget: step-level annotation is roughly an order of magnitude more expensive than outcome labels; MC-labeling trades that cost for rollout compute.
  • Compute budget: PRM-as-RL-reward adds per-step scoring to every training step; budget accordingly.

Default posture: build the verifiable-ORM baseline, measure where it fails, and add a PRM only when the failure mode is specifically bad step-level credit assignment or you need a stronger reranker.

ShareX / TwitterLinkedIn
← Back to Research