What is GRPO and how is it different from PPO?

GRPO (Group Relative Policy Optimization) is a reinforcement-learning algorithm for fine-tuning language models, introduced in the DeepSeekMath paper as "a variant of Proximal Policy Optimization (PPO)." The difference is what plays the role of the baseline. PPO trains a separate value network — a critic, as large as the policy — to predict how good each partial answer is, and subtracts that prediction to get the advantage. GRPO deletes the critic entirely. Instead it samples a group of complete answers to the same prompt and uses the group's own average reward as the baseline. An answer's advantage is simply how far its reward sits above or below its siblings', so there is no second model to train, hold in memory, or keep stable.

How does GRPO compute the advantage?

For each prompt it takes the rewards of its group of generations, subtracts the group mean, and divides by the group standard deviation — a z-score within the group. In the tunix implementation that is three lines: reshape the flat reward vector into (prompts, num_generations), compute the per-group mean and std (with ddof=1, the sample standard deviation), broadcast both back to every generation, and return (rewards − mean) / (std + 1e-6). The tiny 1e-6 in the denominator prevents a divide-by-zero when every answer in a group scored identically — in that case the numerator is zero anyway, so the advantage is zero and the group contributes no gradient.

What is the difference between GRPO, Dr.GRPO, and RLOO?

All three mean-center a group of rewards; they differ in how they scale. GRPO divides the centered reward by the group's standard deviation, so every prompt's advantages are unit-variance. Dr.GRPO ("GRPO done right") drops that division and returns just reward − mean; the std normalization was shown to introduce a length and difficulty bias — easy prompts with tiny spread get their advantages blown up — so removing it makes the update unbiased. RLOO (REINFORCE Leave-One-Out) also skips the std, but computes each answer's baseline as the mean of the *other* answers in the group, excluding itself, dividing by N−1. Algebraically RLOO's advantage is exactly Dr.GRPO's scaled by N/(N−1) — the same centering, seen from the leave-one-out angle.

Why does GRPO not need a value (critic) model?

Because the group supplies the baseline the critic was there to estimate. The advantage in policy-gradient methods is reward minus a baseline; the only requirement for the baseline is that it not depend on the action being scored, so subtracting it lowers variance without adding bias. PPO estimates that baseline per-token with a learned value function. GRPO observes that if you sample several complete answers to the same prompt, the average reward of that group is itself a valid, action-independent baseline — free, no parameters, no training. That halves the model count in the RL loop (policy + reward model instead of policy + reward + value) and removes the value network's own instability, which is a large part of why GRPO is cheaper and steadier than PPO for verifiable-reward tasks like math and code.

What does num_generations mean in GRPO?

num_generations is the group size G — how many separate answers the policy samples for each prompt before computing advantages. The reward vector arrives flat, one reward per generation across all prompts, and the estimator reshapes it to (−1, num_generations) so each row is one prompt's group. Every statistic — the mean, the std, the leave-one-out baseline — is computed within a row, never across prompts. The group must have at least 2 members for a baseline to mean anything: RLOO explicitly returns all-zero advantages when num_generations < 2, because a single sample has no siblings to be measured against. Larger groups give a lower-variance baseline at the cost of more sampling per training step.

GRPO Advantage: Z-Score Your Siblings, Line by Line

PPO learns a whole second neural network just to guess how good an answer is. GRPO throws that away: sample a group of answers to the same prompt, and each answer's advantage is just how far above or below its siblings it scored. We walk the real tunix advantage estimators — GRPO's z-score, Dr.GRPO's un-normalized fix, and RLOO's leave-one-out baseline — three self-contained functions, computed live side by side.

Code walk · AI / ML. The source ↗

A free, interactive, animated visual explainer of GRPO Advantage: Z-Score Your Siblings, Line by Line — built to be understood, not skimmed.

Questions

What is GRPO and how is it different from PPO?: GRPO (Group Relative Policy Optimization) is a reinforcement-learning algorithm for fine-tuning language models, introduced in the DeepSeekMath paper as "a variant of Proximal Policy Optimization (PPO)." The difference is what plays the role of the baseline. PPO trains a separate value network — a critic, as large as the policy — to predict how good each partial answer is, and subtracts that prediction to get the advantage. GRPO deletes the critic entirely. Instead it samples a group of complete answers to the same prompt and uses the group's own average reward as the baseline. An answer's advantage is simply how far its reward sits above or below its siblings', so there is no second model to train, hold in memory, or keep stable.
How does GRPO compute the advantage?: For each prompt it takes the rewards of its group of generations, subtracts the group mean, and divides by the group standard deviation — a z-score within the group. In the tunix implementation that is three lines: reshape the flat reward vector into (prompts, num_generations), compute the per-group mean and std (with ddof=1, the sample standard deviation), broadcast both back to every generation, and return (rewards − mean) / (std + 1e-6). The tiny 1e-6 in the denominator prevents a divide-by-zero when every answer in a group scored identically — in that case the numerator is zero anyway, so the advantage is zero and the group contributes no gradient.
What is the difference between GRPO, Dr.GRPO, and RLOO?: All three mean-center a group of rewards; they differ in how they scale. GRPO divides the centered reward by the group's standard deviation, so every prompt's advantages are unit-variance. Dr.GRPO ("GRPO done right") drops that division and returns just reward − mean; the std normalization was shown to introduce a length and difficulty bias — easy prompts with tiny spread get their advantages blown up — so removing it makes the update unbiased. RLOO (REINFORCE Leave-One-Out) also skips the std, but computes each answer's baseline as the mean of the *other* answers in the group, excluding itself, dividing by N−1. Algebraically RLOO's advantage is exactly Dr.GRPO's scaled by N/(N−1) — the same centering, seen from the leave-one-out angle.
Why does GRPO not need a value (critic) model?: Because the group supplies the baseline the critic was there to estimate. The advantage in policy-gradient methods is reward minus a baseline; the only requirement for the baseline is that it not depend on the action being scored, so subtracting it lowers variance without adding bias. PPO estimates that baseline per-token with a learned value function. GRPO observes that if you sample several complete answers to the same prompt, the average reward of that group is itself a valid, action-independent baseline — free, no parameters, no training. That halves the model count in the RL loop (policy + reward model instead of policy + reward + value) and removes the value network's own instability, which is a large part of why GRPO is cheaper and steadier than PPO for verifiable-reward tasks like math and code.
What does num_generations mean in GRPO?: num_generations is the group size G — how many separate answers the policy samples for each prompt before computing advantages. The reward vector arrives flat, one reward per generation across all prompts, and the estimator reshapes it to (−1, num_generations) so each row is one prompt's group. Every statistic — the mean, the std, the leave-one-out baseline — is computed within a row, never across prompts. The group must have at least 2 members for a baseline to mean anything: RLOO explicitly returns all-zero advantages when num_generations < 2, because a single sample has no siblings to be measured against. Larger groups give a lower-variance baseline at the cost of more sampling per training step.

GRPO Advantage: Z-Score Your Siblings, Line by Line

Questions

Related explainers