What are the different roles in an RL post-training cluster?

A cluster training a policy with reinforcement learning splits the work into five distinct roles, each with its own copy of weights and its own job. The actor is the policy under gradient descent — it owns an optimizer and gets updated every step. The rollout is the same policy used only to sample completions, and may be a literal alias of the actor or a separately-sharded copy behind a faster inference engine. The reference is a frozen snapshot of the pre-RL policy, queried only for log-probabilities so a KL penalty has something honest to measure drift against; it is never trained. The reward scores a completion, whether that is a trained reward model or a plain scoring function. The critic is a value-baseline model used only by PPO-style algorithms — GRPO drops it entirely and estimates the baseline from a group of samples instead.

What is the difference between a colocated and disaggregated RL cluster?

It comes down to one piece of config: a mapping from each role to the accelerator mesh it runs on. In a colocated cluster every role's mesh is the same mesh, and when the actor and rollout share a mesh they can share the literal same weights in memory — no copy, no cross-device traffic to keep them in sync. In a disaggregated cluster the rollout (and sometimes reference and reward) get their own, differently-shaped mesh — often optimized for fast decoding rather than gradient computation — which means every updated weight has to be physically resharded across the interconnect before the sampler can use it. Colocated is cheaper to keep in sync but forces every role to fit in one accelerator fleet at once; disaggregated frees that constraint at the cost of a real data transfer every step.

How does weight sync work between the trainer and the sampler in RL training?

After a gradient step, the trainer takes its parameters — filtered down to just the trainable weights, skipping optimizer state entirely, and to just the LoRA adapter if the policy is LoRA-tuned — and hands them to the rollout engine's update method. That method reshards the parameters onto whatever sharding the rollout copy actually uses. If the rollout shares the actor's mesh, the source and destination shardings are identical and the reshard is a no-op: the same bytes, already in place. If the rollout lives on a different mesh, the same call becomes a genuine cross-device copy, moving every synced byte over the interconnect. It is the same code path either way — the mesh assignment alone decides whether it costs anything.

Why doesn't GRPO need a critic or value model?

Because it gets its baseline for free from the other samples in the same group. Standard policy-gradient methods subtract a learned value estimate from the reward to reduce variance, which means training and maintaining a whole extra model. GRPO instead samples several completions per prompt, and for each one it subtracts the mean reward of that prompt's group and divides by the group's standard deviation — a plain z-score computed directly from the sampled rewards, no critic in sight. A completion that scores above its group's average gets a positive advantage and is reinforced; one that scores below gets pushed down. The trade is real: PPO's critic gives a lower-variance baseline per token, while GRPO's group statistic is coarser but needs no extra model, no extra mesh, and no extra memory.

Why is generating rollouts slower than the training step in RL post-training?

The two phases are bound by different resources. Training runs one big forward-and-backward pass over a full batch — mostly matrix multiplies, which keep the accelerator's compute units busy the whole time. Generating a rollout instead produces one token at a time, and every single token has to stream the entire policy's weights out of memory to produce it — the same bottleneck that makes LLM decoding memory-bandwidth-bound at inference time generally. A rollout engine with no continuous batching or paged attention pays that memory-streaming cost per sequence rather than sharing it across many concurrent sequences, which is exactly why production RL clusters reach for dedicated inference engines for the rollout role instead of reusing the training-shaped model to sample from.

The RL Cluster: Five Roles, One Mesh Dial

An RL policy under training needs five jobs sharing one accelerator fleet: sample completions, score them, hold a frozen baseline, and push a gradient update back — and one line of config decides whether those jobs share memory or ship weights across the wire. We open Google's real tunix RL cluster to see exactly how, and compute what a weight sync costs either way.

System design · AI / ML. The source ↗

A free, interactive, animated visual explainer of The RL Cluster: Five Roles, One Mesh Dial — built to be understood, not skimmed.

Questions

What are the different roles in an RL post-training cluster?: A cluster training a policy with reinforcement learning splits the work into five distinct roles, each with its own copy of weights and its own job. The actor is the policy under gradient descent — it owns an optimizer and gets updated every step. The rollout is the same policy used only to sample completions, and may be a literal alias of the actor or a separately-sharded copy behind a faster inference engine. The reference is a frozen snapshot of the pre-RL policy, queried only for log-probabilities so a KL penalty has something honest to measure drift against; it is never trained. The reward scores a completion, whether that is a trained reward model or a plain scoring function. The critic is a value-baseline model used only by PPO-style algorithms — GRPO drops it entirely and estimates the baseline from a group of samples instead.
What is the difference between a colocated and disaggregated RL cluster?: It comes down to one piece of config: a mapping from each role to the accelerator mesh it runs on. In a colocated cluster every role's mesh is the same mesh, and when the actor and rollout share a mesh they can share the literal same weights in memory — no copy, no cross-device traffic to keep them in sync. In a disaggregated cluster the rollout (and sometimes reference and reward) get their own, differently-shaped mesh — often optimized for fast decoding rather than gradient computation — which means every updated weight has to be physically resharded across the interconnect before the sampler can use it. Colocated is cheaper to keep in sync but forces every role to fit in one accelerator fleet at once; disaggregated frees that constraint at the cost of a real data transfer every step.
How does weight sync work between the trainer and the sampler in RL training?: After a gradient step, the trainer takes its parameters — filtered down to just the trainable weights, skipping optimizer state entirely, and to just the LoRA adapter if the policy is LoRA-tuned — and hands them to the rollout engine's update method. That method reshards the parameters onto whatever sharding the rollout copy actually uses. If the rollout shares the actor's mesh, the source and destination shardings are identical and the reshard is a no-op: the same bytes, already in place. If the rollout lives on a different mesh, the same call becomes a genuine cross-device copy, moving every synced byte over the interconnect. It is the same code path either way — the mesh assignment alone decides whether it costs anything.
Why doesn't GRPO need a critic or value model?: Because it gets its baseline for free from the other samples in the same group. Standard policy-gradient methods subtract a learned value estimate from the reward to reduce variance, which means training and maintaining a whole extra model. GRPO instead samples several completions per prompt, and for each one it subtracts the mean reward of that prompt's group and divides by the group's standard deviation — a plain z-score computed directly from the sampled rewards, no critic in sight. A completion that scores above its group's average gets a positive advantage and is reinforced; one that scores below gets pushed down. The trade is real: PPO's critic gives a lower-variance baseline per token, while GRPO's group statistic is coarser but needs no extra model, no extra mesh, and no extra memory.
Why is generating rollouts slower than the training step in RL post-training?: The two phases are bound by different resources. Training runs one big forward-and-backward pass over a full batch — mostly matrix multiplies, which keep the accelerator's compute units busy the whole time. Generating a rollout instead produces one token at a time, and every single token has to stream the entire policy's weights out of memory to produce it — the same bottleneck that makes LLM decoding memory-bandwidth-bound at inference time generally. A rollout engine with no continuous batching or paged attention pays that memory-streaming cost per sequence rather than sharing it across many concurrent sequences, which is exactly why production RL clusters reach for dedicated inference engines for the rollout role instead of reusing the training-shaped model to sample from.

The RL Cluster: Five Roles, One Mesh Dial

Questions

Related explainers