## GRPO, Simply Explained

GRPO (Group Relative Policy Optimization) is a [reinforcement learning](https://dev.adaptive-ml.com/training/post-training/rl) algorithm for training language models. Most recent RL-on-LLM methods, such as [GSPO](https://dev.adaptive-ml.com/optimization/gspo) and [DAPO](https://dev.adaptive-ml.com/optimization/dapo), are variations on it. Same core loop: generate a group of outputs for a prompt, score them, nudge the model toward what beat average.

### Background

[Supervised fine-tuning](https://dev.adaptive-ml.com/training/post-training/sft) teaches a model from examples: given this prompt, produce this output. Reinforcement learning teaches from [rewards](https://dev.adaptive-ml.com/rewards).

The model being trained is called the policy model. It attempts a task, earns a reward, and learns to earn higher rewards.

A reward tells the model what "good" means. It can come from an [AI judge](https://dev.adaptive-ml.com/rewards/rlaif), a [model trained on human feedback](https://dev.adaptive-ml.com/rewards/reward-models), or [direct verification](https://dev.adaptive-ml.com/rewards/rlvr).

### What GRPO does

Instead of one output per prompt, GRPO produces a group of rollouts. Each rollout is one complete generation from the policy model.

Each rollout gets a reward. The scores are ranked against the group's average.

| Rollout          | Score |
|------------------|-------|
| B·Royal Red      | 0.9   |
| D·Ruby Red       | 0.8   |
| A·Coral Pink     | 0.4   |
| C·Rose Gold      | 0.2   |
| **Average**      | 0.57  |

Each rollout's standardized score is its advantage. It's applied to each of the rollout's tokens.

| Token          | Token Name    | Advantage |
|----------------|----------------|-----------|
| B·Royal Red   | Royal Red      | +1.14     |

Positive advantage makes those tokens more likely. Negative makes them less.

In the training loss, each token's log-probability is weighted by its rollout's advantage.

Calculating loss

For each token in rollout`i`, the loss includes `Ai × log π(token | context)`. Optimizing it pushes the token's probability up when `A` is positive, down when negative.

That's GRPO. The [DeepSeekMath paper](https://arxiv.org/abs/2402.03300) introduces it with the full gradient math. GSPO aggregates the signal at the sequence level instead of the token level, stabilizing training for mixture-of-experts models.

[Want to learn more about RL? Explore the RL Glossary](https://dev.adaptive-ml.com/)