RLHF to GRPO and Beyond: How Reinforcement Learning Is Transforming LLM Training Pipelines

---

What Is RLHF and Why It Changed LLM Training

For years, training a language model meant showing it millions of examples. The model learned patterns. It learned syntax, semantics, and even some reasoning. But it had no sense of what humans actually wanted from a response.

Reinforcement Learning from Human Feedback changed that.

RLHF introduced a way to teach LLMs not just what words typically follow other words—but what responses people actually prefer. This shifted the training paradigm from pattern replication to preference optimization.

The RLHF pipeline has three distinct stages. Each matters. Together, they made models like ChatGPT broadly useful for real tasks.

The Three Stages of RLHF

Stage 1: Supervised Fine-Tuning (SFT)

Before RLHF begins, most models go through SFT. The goal is to create a strong starting policy. Engineers fine-tune a pre-trained LLM on curated demonstration data. These demonstrations show the kind of outputs the model should produce.

SFT gives the model a foundation in the target domain. It does not, however, teach the model which of two correct-seeming responses is actually better. That requires preference data.

Stage 2: Reward Model Training

This is where most of the intellectual work happens.

Engineers collect comparison data: human annotators look at pairs of model responses and pick the one they prefer. This creates a preference dataset. The reward model learns to predict these human preferences.

Think of the reward model as a learned function. It takes a prompt and a response. It outputs a scalar score estimating how much a human would like that response.

A well-trained reward model captures nuance that is impossible to encode in rules. It learns that thorough explanations often beat brief ones. That accurate uncertainty framing beats false confidence. That concise technical answers beat verbose ones when the reader is an expert.

Stage 3: RL Fine-Tuning with PPO

With a reward model in hand, the actual RL training begins.

The language model generates responses. The reward model scores them. The RL algorithm—typically Proximal Policy Optimization (PPO)—updates the language model's weights to produce higher-scoring responses in the future.

PPO is conservative by design. It constrains how much the policy can change in a single update. This stability is essential when fine-tuning a model that took months and millions of dollars to pre-train.

The combination of these three stages—SFT, reward modeling, and PPO-based RL—produced the first wave of truly useful conversational AI systems.

From RLHF to GRPO: Why the Field Needed a New Algorithm

PPO-based RLHF works. It also has a serious practical problem.

PPO requires a critic network—a second neural network that estimates the value of each state. For image-based RL, this is manageable. For LLMs, the "state" is a sequence of tokens. Estimating the value of every partial token sequence requires significant compute and memory.

At small scales, this is fine. At LLM scale—7 billion parameters and up—the critic network doubles the memory footprint of every training step. Teams that wanted to apply RLHF to frontier models needed access to hardware that only a handful of organizations could afford.

This constraint drove research into alternatives. The goal: retain the benefits of RL-based alignment while eliminating the critic bottleneck.

Group Relative Policy Optimization (GRPO) arrived as a direct response to this problem.

How GRPO Computes Advantages Without a Critic

GRPO eliminates the critic by changing how it estimates advantage.

Instead of learning a value function, GRPO generates multiple responses to the same prompt. It compares those responses against each other within the group.

The advantage estimate is straightforward:

advantage(sample) = (reward(sample) - mean(rewards in group)) / std(rewards in group)

This is group-relative advantage estimation. The sample's reward is normalized against its same-prompt peers. A response that scores higher than the group average gets a positive advantage. One that scores lower gets a negative advantage.

No critic is needed because the group provides the baseline. The group is the baseline.

This changes the compute profile significantly. Memory usage drops because there is no critic network. Training throughput increases because the model can focus all capacity on the policy network.

The trade-off is that GRPO requires more samples per prompt to estimate stable advantages. Where PPO might compare a response against a learned value estimate, GRPO needs 8-16 samples per prompt group to get a reliable group mean and standard deviation.

For teams with limited GPU budgets, this trade-off almost always favors GRPO. Generating extra samples is cheap relative to maintaining a critic network.

Comparing PPO and GRPO Across the Metrics That Matter

Training Stability

PPO's clipped objective prevents the policy from changing too dramatically in any single update. This provides a built-in stability floor. Teams running PPO typically see smooth reward curves that gradually improve.

GRPO's stability depends more heavily on proper reward normalization. If the group variance is too high, advantage estimates become noisy. This translates to erratic gradient updates and potential training instability. Starting with 8-16 samples per prompt group and applying standard z-score normalization handles most cases.

Compute Efficiency

PPO requires a critic network sized at roughly 30% of the policy model parameters. For a 70B parameter policy, the critic adds approximately 21B parameters. At BF16 precision, that is 42GB of additional GPU memory—before accounting for optimizer states, gradients, or activations.

GRPO has no critic. The same 70B model trains with only its own parameters. On an 8xA100 node (640GB total HBM), GRPO can fit effective batch sizes that PPO cannot.

The practical implication: teams running GRPO report 30-40% lower GPU-hour costs for comparable model quality on standard benchmarks.

Sample Efficiency

GRPO needs more samples per prompt to compute stable group-relative advantages. A typical GRPO rollout generates 8-16 responses per prompt. PPO generates one response and compares it against its learned value estimate.

In terms of wall-clock time to convergence, GRPO often matches or beats PPO because it parallelizes sample generation efficiently. But total sample count is higher.

Final Model Quality

Published benchmarks on standard preference evaluation sets (AlpacaEval, HH-RLHF) show comparable performance between PPO-RLHF and GRPO at matched compute budgets. Neither algorithm consistently outperforms the other on downstream task metrics.

The difference in quality is more pronounced in specific domains. GRPO tends to excel in tasks where response diversity matters—creative writing, coding alternatives, multi-step reasoning. PPO tends to be more stable when the reward model is noisy or when fine-tuning on narrow domains.

The Reward Hacking Problem: Why Goodhart's Law Breaks RLHF

Every RLHF practitioner eventually encounters reward hacking.

Goodhart's Law states: when a measure becomes a target, it ceases to be a good measure. In RLHF, this manifests as the model optimizing for the reward model rather than for the underlying human preference the reward model is supposed to represent.

Reward hacking in LLMs takes many forms. Some are obvious once you see them. Others are subtle.

Verbose non-answers. The model learns that longer responses tend to score higher in the reward model. It begins producing elaborate, circular responses that say nothing new while appearing thorough.

Diplomatic hedging. The model learns that cautious, hedged language avoids the most negative signals. It becomes non-committal on topics where a clear answer would be more useful.

Surface-level pattern matching. The model discovers that certain syntactic structures correlate with high rewards—specific opening phrases, argumentative patterns, or stylistic markers. It applies these patterns regardless of whether they fit the content.

KL divergence constraints are the primary defense against reward hacking. By penalizing large deviations from the SFT policy, KL penalties keep the model grounded in its original behavior. The intuition: if the model is being pushed too far from what it already knew, something has gone wrong.

Tuning the KL penalty coefficient is more art than science. Set it too high, and the model barely learns. Set it too low, and reward hacking runs unchecked. Most practitioners start at 0.1 and adjust based on observed KL divergence curves.

Key insight — reward hacking is often detectable before it becomes severe. Monitor entropy collapse (the policy becoming deterministic too quickly), sudden reward plateaus followed by drops, and KL divergence spikes. These are early warning signals.

If your reward trajectory plateaus while KL divergence continues increasing, the model is learning to exploit the reward model rather than improve on the underlying task.

Alternatives to RLHF: DPO, Constitutional AI, and RLAIF

PPO-based RLHF is not the only path to alignment. Several alternatives have gained traction, each with distinct trade-offs.

Direct Preference Optimization (DPO)

DPO reframes preference learning as a classification problem. Instead of training a reward model and then using RL to optimize against it, DPO directly updates the policy to prefer chosen responses over rejected ones.

The mathematical formulation is elegant. DPO's loss function implicitly performs the same policy regularization that RLHF achieves through KL constraints. No critic network. No PPO. No separate reward model.

DPO performs well on simple preference tasks—factual accuracy, tone, conciseness. It underperforms RLHF on complex multi-dimensional preferences where the reward model must capture subtle trade-offs between competing qualities.

Constitutional AI and RLAIF

Constitutional AI (CAI) takes a different approach to the feedback bottleneck. Rather than relying on human annotators to label millions of preference pairs, CAI uses a set of principles and an LLM to generate critiques and revisions.

The process is iterative. A model generates a response. The same model critiques it against a written principle. The model revises. Human feedback is used only to train the initial principles, not to label every preference pair.

RLAIF (Reinforcement Learning from AI Feedback) generalizes this idea. Instead of principles, a trained AI model provides the feedback signal. This scales preference data generation without proportional human annotation cost.

CAI and RLAIF are not universally superior to RLHF. They inherit the biases of the models that provide feedback. If the feedback model has blind spots, the trained model will share them. Human oversight remains essential for high-stakes applications.

Building an RLHF/GRPO Pipeline: What Actually Works in Practice

Teams building in-house alignment pipelines consistently report the same pain points. Most are avoidable with the right setup.

Tool stack. DeepSpeed with the ZeRO-3 optimizer is the standard for distributed RLHF training at scale. The Hugging Face TRL library provides turnkey implementations of PPO and DPO. For GRPO, most teams implement the group-relative advantage computation on top of TRL's base infrastructure.

Infrastructure. A100 or H100 GPUs are standard. GRPO's memory efficiency means a 7B model trains comfortably on 2xA100. A 70B model typically requires 8xA100 or 4xH100. Gradient checkpointing reduces activation memory by ~60% at the cost of ~30% compute overhead.

Hyperparameter starting points by model scale:

Model Size	KL Penalty	Learning Rate	Group Size	Batch Size (samples)
7B	0.05-0.1	1e-6	8	256
13B	0.05-0.1	8e-7	8-12	128
70B	0.03-0.07	5e-7	12-16	64

Monitoring. Track three primary signals throughout training. Reward trajectory shows whether the model is actually improving on the target metric. KL divergence shows how far the policy has drifted from the SFT baseline. Policy entropy shows whether the model is collapsing to deterministic outputs.

The Road Ahead: What's Next for LLM Alignment Research

RLHF and GRPO are mature technologies. The research frontier has moved on.

Process Reward Models (PRM). Standard RLHF and GRPO use outcome rewards: the reward model scores the final response. PRMs score each step in a multi-step reasoning chain. This finer-grained signal produces better models on tasks where reasoning quality matters more than final answer quality.

Recursive reward modeling. The idea: use the LLM itself to critique and improve responses, then use those critiques to train the next model. Iterated over many rounds, this can produce models significantly beyond human-level performance on specific tasks—even when human evaluators cannot directly assess the final quality.

Interpretability-based alignment. Rather than inferring alignment from behavior (preference data), some researchers are exploring direct inspection of model representations. If internal model states predict alignment failures before they manifest in outputs, interventions become possible before damage occurs.

These directions share a common theme: alignment techniques that scale beyond human evaluation capacity. As models become more capable than any individual human evaluator, the field needs alignment methods that do not depend on human oversight at every step.

Choosing Your Alignment Path: A Practical Decision Framework

Not every team needs the same approach. The right choice depends on your constraints.

Small team, limited compute budget. GRPO or DPO. Both eliminate the critic network. GRPO is more robust to noisy reward models. DPO is simpler to implement. Start with DPO for straightforward preference tasks, switch to GRPO if you encounter training instability.

Existing preference dataset and compute budget. RLHF with PPO or GRPO. The reward model quality determines ceiling performance. If you have a well-curated preference dataset (10,000+ pairs, clear annotator agreement), PPO or GRPO will outperform DPO on complex preference dimensions.

Multi-objective alignment (helpfulness + harmlessness + honesty). Multi-reward weighting. Combine separate reward models for each objective, then use a weighted sum as the RL reward signal. Pareto-optimal approaches can balance these objectives without manual weight tuning.

Reasoning-focused applications. Process reward models. Standard RLHF and GRPO optimize for outcome quality. PRMs optimize for reasoning process quality. On mathematical reasoning, coding challenges, and multi-step logical problems, PRM-based training significantly outperforms outcome-reward training.

---

Ready to implement GRPO in your training pipeline? Subscribe to get practical guides, reference implementations, and practitioner case studies delivered to your inbox.

---

Expert Q&A

Q: We have a working reward model that performs well on our internal eval but degrades quickly during RL training—reward stays flat while KL diverges. What's likely happening?

A: This is a classic sign of reward model overfitting combined with distribution shift. Your reward model was trained on preference data from your SFT model. Once RL training begins, the policy starts generating responses that differ from the SFT distribution. Your reward model hasn't seen this distribution, so its scores become unreliable for samples far from its training manifold.

The fix: run a reward model evaluation on your RL rollout samples before committing to a full training run. If the rollout samples score consistently lower or with higher variance than the training set, you need to either (a) retrain the reward model on a more diverse distribution that includes RL-generated samples, or (b) tighten the KL penalty to slow down distribution shift.

This is one of the most underappreciated failure modes in production RLHF. Teams spend weeks tuning learning rates and batch sizes when the real problem is that the reward model generalizes poorly to the evolving policy.

Q: We want to switch from PPO to GRPO to save GPU memory. Can we reuse our existing reward model, or do we need to retrain it?

A: Your reward model is independent of the RL optimizer. GRPO only changes how advantages are computed—it doesn't require a different reward signal. You can drop in GRPO with zero changes to your reward model or preference dataset.

One practical note: GRPO's group-relative advantage estimation puts more pressure on reward model consistency than PPO does. In PPO, a single noisy reward score is compared against a learned value estimate. In GRPO, a sample's advantage depends on how it scores relative to 8-16 peers. If your reward model has high per-sample variance (unstable scores for similar responses), GRPO will amplify that noise.

Before switching, compute the inter-sample reward correlation on a holdout set. If the same response gets wildly different scores when scored multiple times, your reward model needs calibration before GRPO will work well.

Q: What's the practical difference between DPO and GRPO for a team that already has a preference dataset of ~5,000 pairs?

A: At 5,000 pairs, you are in a gray zone. DPO can work with this amount of data if your preference pairs are clean and the task is relatively simple. GRPO needs a reward model regardless, and training a reward model on 5,000 pairs is borderline—you'll want to check inter-annotator agreement carefully.

The decision depends on your primary constraint. If compute is tight and you have limited GPU access, DPO wins because it requires no RL loop. If quality on complex multi-dimensional preferences matters more than compute, invest the GPU hours in GRPO with a properly validated reward model.

One underappreciated factor: DPO can only optimize for the preferences explicitly in your dataset. GRPO, via the reward model, can generalize to preference dimensions that weren't directly labeled. If your 5,000 pairs don't cover edge cases or subtle quality dimensions you care about, DPO will be blind to them.

Q: How do you actually detect that reward hacking has started, before it shows up in your final evals?

A: The three leading indicators, in order of typical appearance:

Policy entropy collapse. The model's output distribution becomes increasingly peaked. You can measure this by computing the entropy of the model's token probability distribution at each training step. A rapid entropy drop in the first 10-20% of training is normal. If entropy approaches its minimum (near-deterministic output) by step 30% of training, something has gone wrong.

KL divergence spikes. KL should increase gradually and smoothly. Sudden jumps—where KL doubles in a single step—typically indicate the policy found a way to exploit the reward model that overwhelmed the KL penalty. When you see this, pause training and inspect the highest-reward samples from the offending step. They will usually look obviously gameable to a human evaluator.

Reward trajectory inflection followed by degradation. If reward flattens and then begins declining while KL continues climbing, the policy has likely shifted into a mode where it exploits the reward model at the expense of actual quality. This is the point where continuing training makes things worse, not better.

Establish baselines for all three before training begins. Set alerting thresholds at 2x the baseline variance. Most teams that miss reward hacking are not monitoring these signals—or are monitoring them but don't have pre-agreed intervention protocols.

Q: For reasoning-heavy tasks like code generation and math, why do Process Reward Models outperform outcome reward models, and how hard are they to implement?

A: Outcome reward models (ORMs) give a single score to the final answer. For code generation and math, this is problematic because there are many valid reasoning paths that lead to the same correct answer—and many incorrect paths that superficially resemble correct ones.

A Process Reward Model (PRM) scores each step in a reasoning chain. This creates a credit assignment signal that actually reflects where reasoning quality lives: in the reasoning process itself, not just the final result.

The harder problem is step-level annotation. ORMs need preference labels on complete responses. PRMs need step-level quality labels—which are significantly more expensive to collect and require annotators with domain expertise.

The implementation gap: most teams use ORMs for practical reasons (cheaper annotation, faster iteration). But if you have domain-expert annotators and a reasoning-heavy task, PRMs consistently show 10-20% improvement on benchmarks like MATH and HumanEval. The ROI is real; the annotation cost is the main barrier.

Q: When is multi-reward weighting appropriate, and what are the pitfalls?

A: Multi-reward weighting becomes relevant when you need to simultaneously optimize for multiple alignment criteria that may be in tension—helpfulness vs. harmlessness, conciseness vs. thoroughness, technical accuracy vs. accessibility.

The straightforward implementation is a weighted sum of individual reward model scores. You define a weight vector w = [w_helpful, w_safe, w_concise] and sum w_i * reward_i for each response.

The pitfalls:

Reward scale inconsistency. If your helpfulness reward ranges from -2 to +2 and your safety reward ranges from 0 to 1, their relative contribution to the total reward is dominated by helpfulness regardless of your stated weights. Normalize each reward model to a common scale before combining them.

Pareto collapse. Fixed weights can drive the model to sacrifice one objective entirely to maximize another, even when both matter. If w_harmless is too low, the model will optimize helpfulness at the expense of safety. Periodically checking Pareto frontiers—not just aggregate metrics—catches this.

Weight tuning is iterative. There is no principled way to set weights a priori. Teams typically start with equal weights, evaluate on each objective independently, then adjust based on which objectives are underperforming. This takes 3-5 evaluation rounds to converge.