What is post-training and why does a base model need it?

Post-training is everything you do to a pretrained language model after next-token pretraining to make it useful: supervised fine-tuning on demonstrations, then preference-based alignment (RLHF, DPO, or GRPO). A base model has only ever learned to continue text — it has no notion of an instruction or of being helpful. Ask it a question and it may answer, or it may write three more questions, because "continue this document" is all it was trained to do. Post-training is what turns that raw predictor into an assistant that follows instructions and refuses harmful ones.

What is the difference between SFT, RLHF, and DPO?

SFT (supervised fine-tuning) trains the model to imitate human-written demonstrations of good answers — it teaches the format and the behavior but only from positive examples. RLHF (reinforcement learning from human feedback) goes further: it trains a reward model on human preference comparisons, then uses reinforcement learning (usually PPO) to push the policy toward higher-reward outputs while a KL penalty keeps it close to the SFT model. DPO (Direct Preference Optimization) reaches the same objective without training a separate reward model or running an RL loop — it reparameterizes the reward so the preferences can be optimized directly with "a simple classification loss".

How is a reward model trained in RLHF?

On pairwise human preferences. Labelers are shown two model responses to the same prompt and pick the better one; the reward model — usually the SFT model with the output head swapped for a scalar — is trained so the preferred response scores higher than the rejected one. Under the Bradley-Terry model this is a logistic loss on the score difference: the probability that response A beats B is the sigmoid of (reward_A − reward_B). The trained reward model turns a subjective human judgment into a single scalar the RL step can maximize.

Why is PPO-based RLHF expensive and unstable?

Because it keeps up to four models involved per batch. The policy generates completions; a frozen reference copy of the SFT model scores the same tokens to compute the KL penalty; the reward model scores the completion; and a value (critic) model estimates a per-token baseline for the advantage. Three of those (policy, reference, reward) and often a fourth (value) sit in memory at once. The KL penalty is a leash — "the KL divergence term penalizes the RL policy from moving substantially away from the initial pretrained model" — and if it is too loose the policy drifts and reward-hacks; too tight and it never improves. DPO and GRPO are largely responses to this cost and instability.

When should you use DPO vs PPO vs GRPO?

DPO is the default when you have a fixed dataset of preference pairs and want a stable, cheap, offline recipe — no reward model, no sampling loop, "computationally lightweight". PPO still earns its cost when you want online exploration against a reward signal that can be re-queried, or a reward model you keep improving in the loop. GRPO, introduced in DeepSeekMath as "a variant of Proximal Policy Optimization (PPO)", drops the value model and estimates the baseline from a group of sampled answers instead, cutting PPO’s memory — it shines when rewards are verifiable (math, code) and you can afford to sample many completions per prompt.

The Post-Training Pipeline (SFT → RLHF → DPO)

A base model only predicts the next token — it has never been told to be helpful. Post-training is how it learns: supervised fine-tuning on demonstrations, a reward model distilled from pairwise preferences, then RLHF where four models fight in memory on a KL leash — or DPO, which folds the reward model into a single classification loss. We draw the reward curve, animate the four-model dance per batch, watch a policy hack a length-biased reward, and lay SFT/DPO/PPO/GRPO flat on a trade-off table.

System design · AI / ML. The source ↗

A free, interactive, animated visual explainer of The Post-Training Pipeline (SFT → RLHF → DPO) — built to be understood, not skimmed.

Questions

What is post-training and why does a base model need it?: Post-training is everything you do to a pretrained language model after next-token pretraining to make it useful: supervised fine-tuning on demonstrations, then preference-based alignment (RLHF, DPO, or GRPO). A base model has only ever learned to continue text — it has no notion of an instruction or of being helpful. Ask it a question and it may answer, or it may write three more questions, because "continue this document" is all it was trained to do. Post-training is what turns that raw predictor into an assistant that follows instructions and refuses harmful ones.
What is the difference between SFT, RLHF, and DPO?: SFT (supervised fine-tuning) trains the model to imitate human-written demonstrations of good answers — it teaches the format and the behavior but only from positive examples. RLHF (reinforcement learning from human feedback) goes further: it trains a reward model on human preference comparisons, then uses reinforcement learning (usually PPO) to push the policy toward higher-reward outputs while a KL penalty keeps it close to the SFT model. DPO (Direct Preference Optimization) reaches the same objective without training a separate reward model or running an RL loop — it reparameterizes the reward so the preferences can be optimized directly with "a simple classification loss".
How is a reward model trained in RLHF?: On pairwise human preferences. Labelers are shown two model responses to the same prompt and pick the better one; the reward model — usually the SFT model with the output head swapped for a scalar — is trained so the preferred response scores higher than the rejected one. Under the Bradley-Terry model this is a logistic loss on the score difference: the probability that response A beats B is the sigmoid of (reward_A − reward_B). The trained reward model turns a subjective human judgment into a single scalar the RL step can maximize.
Why is PPO-based RLHF expensive and unstable?: Because it keeps up to four models involved per batch. The policy generates completions; a frozen reference copy of the SFT model scores the same tokens to compute the KL penalty; the reward model scores the completion; and a value (critic) model estimates a per-token baseline for the advantage. Three of those (policy, reference, reward) and often a fourth (value) sit in memory at once. The KL penalty is a leash — "the KL divergence term penalizes the RL policy from moving substantially away from the initial pretrained model" — and if it is too loose the policy drifts and reward-hacks; too tight and it never improves. DPO and GRPO are largely responses to this cost and instability.
When should you use DPO vs PPO vs GRPO?: DPO is the default when you have a fixed dataset of preference pairs and want a stable, cheap, offline recipe — no reward model, no sampling loop, "computationally lightweight". PPO still earns its cost when you want online exploration against a reward signal that can be re-queried, or a reward model you keep improving in the loop. GRPO, introduced in DeepSeekMath as "a variant of Proximal Policy Optimization (PPO)", drops the value model and estimates the baseline from a group of sampled answers instead, cutting PPO’s memory — it shines when rewards are verifiable (math, code) and you can afford to sample many completions per prompt.

The Post-Training Pipeline (SFT → RLHF → DPO)

Questions

Related explainers