AI Researchprocess reward modelsoutcome reward modelsreasoning model trainingPRM vs ORM

Process Reward Models vs Outcome Rewards: Why Reasoning Training Split in Two

Compare process reward models and outcome rewards for reasoning training. Learn strengths, failure modes, and how to pick the right approach for your team.

The Split in Reasoning Training

The question engineering leaders are asking

If you are building with reasoning models, you have noticed the field has quietly forked. Two reward paradigms now dominate how labs train models to think: outcome rewards that grade only the final answer, and process reward models (PRMs) that grade each step along the way. The choice shapes model quality, evaluation design, and your infrastructure bill.

The fork is visible in the research record. Outcome-based training scaled first because it rode on infrastructure that already existed. Process-based training followed because outcome signals alone hit a ceiling on long, multi-step problems. OpenAI's "Let's Verify Step by Step" work (Lightman et al., 2023) made the case empirically: on a 500-problem subset of the MATH benchmark, a process-supervised verifier beat an outcome-supervised one, and the advantage grew with the number of candidates used for reranking (best-of-N). The same paper reported a null result on the APPS coding task — process supervision did not improve over outcome supervision there — a detail worth remembering when you pick your domain. The open PRM line that followed (PRM800K, Math-Shepherd, and later Qwen2.5-Math-PRM) pushed the idea toward production, and the public research record shows labs increasingly combining both paradigms rather than picking one. (See our primer on how reasoning models are trained.)

The timing tells you something about why the split happened when it did. Outcome-based RL fine-tuning of reasoning models became practical around 2022–2023, when teams realized that existing RLHF stacks could be repointed at verifiable tasks without rebuilding the labeling pipeline. Process supervision existed as an idea well before that — the insight that intermediate steps carry more learning signal than endpoints is old — but it required a new kind of data that nobody had at scale. The fork, in other words, was driven less by a theoretical disagreement than by which supervision signal was cheap enough to produce first.

What this brief covers

This article explains what each reward type optimizes, how the two are trained, and where each breaks down. It closes with a decision framework for task selection and a look at hybrid methods. There is no single winner. The right answer depends on your domain, your budget, and how much you can afford to trust a verifier. (Related: reward hacking and how to detect it.)

Side-by-side diagram of an ORM scoring only the final answer versus a PRM scoring each reasoning ste
Side-by-side diagram of an ORM scoring only the final answer versus a PRM scoring each reasoning ste

What Outcome Reward Models Optimize For

Final-answer supervision

An outcome reward model (ORM) scores a complete trajectory by its endpoint. For math, that means checking the final number against a known answer. For code, it means running the test suite. The signal is binary or scalar and arrives once, at the end.

This is why ORMs scaled so fast. Existing preference and RLHF pipelines already produce outcome labels, and any domain with a checker (unit tests, symbolic solvers, ground-truth labels) generates them for free. You do not need annotators to read a chain of thought. A codebase with a CI pipeline is, in effect, an outcome-label factory. A math dataset with answer keys is the same.

The economics are hard to beat. Outcome labels cost roughly nothing per trajectory once the checker exists, and they are objective — the test either passes or it does not. That objectivity is why ORMs became the default for the first wave of reasoning RL. (See RLHF pipeline basics.)

The training loop is also simple to operate. You generate rollouts, run the checker, and feed the resulting scalar rewards into a policy-gradient method such as PPO or GRPO. There is no second model to keep in sync, no step-alignment problem, and no question of whether the reward model has drifted away from the policy's current behavior. For a team shipping its first reasoning fine-tune, that operational simplicity is a real advantage — the failure modes are mostly the model's, not the pipeline's.

Strengths and failure modes

ORMs are cheap to supervise, easy to automate, and robust when the answer is verifiable. Their weakness is sparse credit assignment. When a 40-step trajectory fails, the reward says nothing about which step went wrong. The model must infer blame from a single signal, which slows learning and encourages shortcuts.

The deeper problem is reward hacking. A model can produce the right answer through wrong reasoning and still collect the reward. That reasoning does not generalize, so benchmark scores rise while real-world reliability lags. This is the "right answer, wrong reasoning" gap that outcome-only training struggles to close.

A concrete failure mode: train a model on competition math with answer-only rewards, and it learns to pattern-match problem templates rather than reason. Pass rates on the training distribution climb; transfer to held-out problem types stalls. The same dynamic shows up in code, where a model learns to satisfy the visible test cases — hardcoding expected outputs, special-casing inputs — without implementing the actual algorithm. The reward is satisfied, the software is broken, and the failure only surfaces when the hidden test suite runs. (Related: evaluating reasoning models beyond benchmarks.)

A second failure mode is variance. Because the reward arrives once per trajectory, a single unlucky step can sink an otherwise sound chain of thought, and the model receives the same zero whether it made one small slip or went wrong at step three. On long problems this makes the gradient noisy and the learning curve lumpy. Teams often respond by generating many rollouts per prompt to average out that noise, which quietly raises the compute cost of "cheap" outcome labels.

Outcome rewards tell you whether the model succeeded. They rarely tell you why — and "why" is what transfers to the next problem.

What Process Reward Models Optimize For

Step-level credit assignment

A process reward model (PRM) assigns credit at intermediate steps. Instead of one label per trajectory, you get a score per reasoning step, giving dense feedback that pinpoints where a chain of thought derails. This is valuable for three concrete jobs: guiding search, reranking candidate solutions, and catching errors early before they compound.

The math of dense versus sparse signal matters. With a binary outcome reward, a 40-step trajectory yields one bit of information. With step-level scores, the same trajectory yields dozens of graded signals, and the policy can update on the specific step that failed rather than the whole chain. This can improve sample efficiency — fewer rollouts to reach a given accuracy — but only when the step labels are high quality. A noisy or miscalibrated PRM injects its own errors into the gradient, and the efficiency gain evaporates. Treat the denser signal as a potential lever, not a guaranteed one.

PRMs also make better verifiers. A step-level scorer can rank multiple candidate trajectories at inference time, so a smaller model with a strong PRM can outperform a larger model generating a single pass. For teams optimizing cost per correct answer, that reranking layer is often the highest-leverage addition. Generate N candidates cheaply, score each step with the PRM, keep the trajectory with the highest cumulative step score — this is best-of-N with a process-aware selector, and with a well-calibrated verifier it can beat naive majority voting. With a weak one, it can lose to it. Measure on your own distribution before you commit. (See best-of-N sampling and verifier reranking.)

There is a practical asymmetry here worth internalizing: reranking is a much lower-risk use of a PRM than training against it. When the PRM only selects among candidates, a miscalibrated score costs you a suboptimal pick. When the PRM is the training reward, a miscalibrated score shapes the policy itself, and the policy will drift toward whatever the verifier rewards. Many teams get most of the benefit by deploying the PRM as a selector first and only promoting it to a training signal after it has proven calibrated on held-out data.

Verifier models and the labeling bottleneck

The catch is supervision. Step-level human annotation is expensive and subjective — annotators disagree about whether a step is "correct" versus merely "plausible." Quality control is hard, and labels do not scale the way test suites do. Published PRM datasets like PRM800K required tens of thousands of human step labels, and inter-annotator agreement on "is this step correct" is meaningfully lower than agreement on "is this final answer correct."

The common alternatives are to generate step labels automatically — Monte Carlo rollouts that estimate the value of each partial solution, or distillation from a stronger model — and both trade annotation cost for label noise and compute. Neither is free.

Math-Shepherd-style automatic labeling, for example, estimates the value of a partial solution by completing it many times with the policy and measuring how often the completions reach a correct final answer. That converts a step-labeling problem into a rollout-budget problem: more completions per step means less noise but more compute. Distillation from a stronger model is cheaper per label but inherits the teacher's biases, including its own blind spots. Whichever route you take, the label supply chain becomes part of your infrastructure, and its quality determines whether the PRM helps or hurts.

The verifier is also hackable

A PRM is a model, and a policy trained against it will learn to satisfy the verifier, not the truth. The failure mode mirrors ORM reward hacking one level up: the model produces steps that look plausible to the scorer — confident tone, standard notation, familiar intermediate forms — while the actual logic is wrong. Because the verifier is the reward, its blind spots become the policy's training target. This is why PRM quality control (held-out step labels, adversarial probes, monitoring for distribution shift between the policy and the verifier's training data) is not optional. A PRM you have not stress-tested is a liability, not a safety net. (Related: reward hacking and how to detect it.)

The distribution-shift point deserves emphasis. A PRM trained on human-written or strong-model solutions may score the policy's own style of reasoning poorly, not because the reasoning is wrong but because it looks unfamiliar. As training proceeds and the policy drifts, the verifier's calibration on the policy's outputs decays. Periodic re-scoring of fresh policy rollouts against held-out ground truth is the cheapest way to catch this before it compounds.

Decision tree mapping task verifiability and reasoning length to ORM, PRM, or hybrid selection
Decision tree mapping task verifiability and reasoning length to ORM, PRM, or hybrid selection

Choosing Between Them: A Decision Framework

SignalLean outcome rewardLean process reward
Verifier exists (tests, solver, answer key)YesOptional
Task length / step countShortLong, multi-step
Label cost toleranceLowHigh (or automate labels)
Failure cost of wrong reasoningLowHigh (regulated, safety-critical)
Inference budget for a second modelNoneAvailable
Tolerance for verifier bias—Manageable with QC

Two cost realities to weigh honestly:

  • Outcome labels are near-free once a checker exists; PRM labels are not. Budget for annotation, automatic label generation, or distillation, plus ongoing QC.
  • A PRM adds a model to the inference loop. Reranking N candidates with a step scorer costs more compute per query than a single pass. The "cheaper per correct answer" case only holds if the accuracy gain outweighs that overhead on your workload — so instrument it.

A third, quieter cost is organizational. Outcome pipelines can be owned by whoever owns the test suite. PRM pipelines need a labeling or synthetic-data function, a verifier-training function, and a monitoring function watching for drift. That is more surface area to staff and more places for a regression to hide. Factor it in before committing to the denser signal.

Hybrid methods

The pragmatic default for most teams is neither pure paradigm. Combine a cheap outcome signal (tests, answer keys) with a PRM used primarily as an inference-time reranker rather than the sole training reward. This keeps the objective grounded in verifiable truth while using dense scores where they add the most value: selecting among candidates and flagging likely-bad trajectories for review. Where step labels are

ShareX / TwitterLinkedIn
← Back to Research