Why is LLM decoding memory-bandwidth-bound instead of compute-bound?

Because generating each new token reads far more bytes than it does arithmetic. Prefill processes the whole prompt at once — a big matrix multiply that keeps the tensor cores busy, so it is compute-bound. Decode then emits one token at a time, and every step must stream the entire model weights (and the whole KV cache of every active sequence) out of HBM to produce a single column of output. As one analysis puts it, "Prefill is a matrix multiplication problem. Decode is a memory bandwidth problem" — the tensor cores finish in microseconds and then wait on the next memory read. So a single decode stream is capped near HBM bandwidth ÷ model bytes, which is why batching many streams together (they share one weight read) is the only way to use the GPU well.

What is the KV cache and why does it grow with context length?

The KV cache stores the key and value vectors that attention has already computed for every previous token, so each new token can attend to the past without recomputing it. Its size is exactly linear in the number of tokens: KV bytes = 2 (key and value) × layers × (kv-heads × head-dim) × bytes-per-element × tokens. For a 13B model at FP16 that is roughly 0.8 MB per token, so a single 2,048-token sequence needs about 1.6 GB — vLLM measured "up to 1.7GB for a single sequence in LLaMA-13B." Because it grows per token and per concurrent request, the KV cache — not the weights — is usually what caps how many streams a GPU can serve.

What is continuous batching in LLM serving?

Continuous batching (also called in-flight or iteration-level batching) rebuilds the batch on every token step instead of once per request. With static batching the whole batch waits for its longest sequence to finish, and "the GPU will be underutilized until the last sequence in the batch finishes generation." Continuous batching frees each slot the moment its sequence emits an end-of-sequence token and pulls a waiting request into it immediately — "Once a sequence emits an end-of-sequence token, we insert a new sequence in its place." Because short and long requests no longer block each other, the GPU stays near-full; Anyscale measured "up to 23x throughput improvement using continuous batching," and vLLM reports up to 24x over naive serving.

What problem does PagedAttention solve?

Fragmentation and over-reservation of KV cache memory. A naive server reserves a contiguous block big enough for each request's maximum possible length, so most of that memory sits empty — vLLM found "existing systems waste 60% – 80% of memory due to fragmentation and over-reservation." PagedAttention, "an attention algorithm inspired by the classic idea of virtual memory and paging in operating systems," splits the KV cache into fixed-size blocks that need not be contiguous, so waste "only happens in the last block of a sequence" — under 4%. More usable memory means more concurrent sequences in the batch, which is more throughput.

What is prefill/decode disaggregation?

It is splitting the two phases of inference onto different GPU pools because they stress different resources. Prefill is compute-bound and bursty; decode is memory-bandwidth-bound and long-running, and mixing them on one GPU means a long prompt's prefill stalls everyone else's decode (head-of-line blocking). Disaggregation runs prefill on one set of GPUs and streams the resulting KV cache to a separate decode pool, so each phase is scheduled and scaled for its own bottleneck. It trades a KV-cache transfer over the interconnect for predictable time-to-first-token and steady inter-token latency — the design behind systems like DistServe and Mooncake.

Design an LLM Serving Platform

One trained model, thousands of concurrent chat and batch streams, and a GPU that costs by the hour. Start from the roofline that rules everything — prefill is compute-bound, decode is memory-bandwidth-bound — then build up through the KV cache, continuous batching, paged attention, and prefill/decode disaggregation. Drawn, computed, and animated.

System design · AI / ML. The source ↗

A free, interactive, animated visual explainer of Design an LLM Serving Platform — built to be understood, not skimmed.

Questions

Why is LLM decoding memory-bandwidth-bound instead of compute-bound?: Because generating each new token reads far more bytes than it does arithmetic. Prefill processes the whole prompt at once — a big matrix multiply that keeps the tensor cores busy, so it is compute-bound. Decode then emits one token at a time, and every step must stream the entire model weights (and the whole KV cache of every active sequence) out of HBM to produce a single column of output. As one analysis puts it, "Prefill is a matrix multiplication problem. Decode is a memory bandwidth problem" — the tensor cores finish in microseconds and then wait on the next memory read. So a single decode stream is capped near HBM bandwidth ÷ model bytes, which is why batching many streams together (they share one weight read) is the only way to use the GPU well.
What is the KV cache and why does it grow with context length?: The KV cache stores the key and value vectors that attention has already computed for every previous token, so each new token can attend to the past without recomputing it. Its size is exactly linear in the number of tokens: KV bytes = 2 (key and value) × layers × (kv-heads × head-dim) × bytes-per-element × tokens. For a 13B model at FP16 that is roughly 0.8 MB per token, so a single 2,048-token sequence needs about 1.6 GB — vLLM measured "up to 1.7GB for a single sequence in LLaMA-13B." Because it grows per token and per concurrent request, the KV cache — not the weights — is usually what caps how many streams a GPU can serve.
What is continuous batching in LLM serving?: Continuous batching (also called in-flight or iteration-level batching) rebuilds the batch on every token step instead of once per request. With static batching the whole batch waits for its longest sequence to finish, and "the GPU will be underutilized until the last sequence in the batch finishes generation." Continuous batching frees each slot the moment its sequence emits an end-of-sequence token and pulls a waiting request into it immediately — "Once a sequence emits an end-of-sequence token, we insert a new sequence in its place." Because short and long requests no longer block each other, the GPU stays near-full; Anyscale measured "up to 23x throughput improvement using continuous batching," and vLLM reports up to 24x over naive serving.
What problem does PagedAttention solve?: Fragmentation and over-reservation of KV cache memory. A naive server reserves a contiguous block big enough for each request's maximum possible length, so most of that memory sits empty — vLLM found "existing systems waste 60% – 80% of memory due to fragmentation and over-reservation." PagedAttention, "an attention algorithm inspired by the classic idea of virtual memory and paging in operating systems," splits the KV cache into fixed-size blocks that need not be contiguous, so waste "only happens in the last block of a sequence" — under 4%. More usable memory means more concurrent sequences in the batch, which is more throughput.
What is prefill/decode disaggregation?: It is splitting the two phases of inference onto different GPU pools because they stress different resources. Prefill is compute-bound and bursty; decode is memory-bandwidth-bound and long-running, and mixing them on one GPU means a long prompt's prefill stalls everyone else's decode (head-of-line blocking). Disaggregation runs prefill on one set of GPUs and streams the resulting KV cache to a separate decode pool, so each phase is scheduled and scaled for its own bottleneck. It trades a KV-cache transfer over the interconnect for predictable time-to-first-token and steady inter-token latency — the design behind systems like DistServe and Mooncake.

Design an LLM Serving Platform

Questions

Related explainers