What are the three numbers that decide GPU workloads?

Every ML-infra estimate reduces to three quantities: the FLOPs a computation costs, the bytes it must move between memory and the compute units, and the bytes it must store. FLOPs set the compute floor, bytes-moved set the memory-bandwidth floor, and bytes-stored decide whether the thing fits on the GPUs you have. The actual run time of any kernel is the larger of (FLOPs ÷ peak compute) and (bytes ÷ memory bandwidth) — whichever wall you hit first. Almost every other figure you care about (throughput, cost, how many GPUs, how long) derives from these three, which is why estimating them well is the whole game.

What is arithmetic intensity and the roofline model?

Arithmetic intensity is the ratio of work to traffic — FLOPs performed per byte moved from memory (op/byte). The roofline model, from Williams, Waterman and Patterson (2009), plots attainable performance against arithmetic intensity on log-log axes: a slanted line (memory bandwidth × intensity) rising until it meets a flat ceiling (the chip's peak compute). Where they meet is the ridge point, equal to peak-FLOPs ÷ bandwidth — the machine balance. A workload whose intensity sits left of the ridge is memory-bound (starved for bandwidth, compute idle); right of the ridge it is compute-bound. Reading which side you land on tells you whether faster math or faster memory would help — and usually it is memory.

What is the 6ND rule for training FLOPs?

Training a dense transformer with N parameters on D tokens costs roughly 6ND floating-point operations. It comes from the matmul cost 2mnk: a forward pass touches every parameter once as a multiply-add per token, about 2N FLOPs per token; the backward pass computes gradients with respect to both the inputs and the weights, roughly twice the forward work, about 4N per token. Two plus four is six, so total training compute ≈ 6 × N × D. For Llama-3 8B on over 15 trillion tokens that is 6 × 8e9 × 15e12 ≈ 7.2e23 FLOPs — a single multiplication that lets you size a training run before you rent a single GPU.

What is MFU and why can't GPUs hit peak FLOPs?

MFU (Model FLOPs Utilization) is the fraction of a GPU's advertised peak compute that a real training run actually sustains — realized FLOPs ÷ peak FLOPs. Nobody hits 100%. The headline TFLOPS on a datasheet is a dense tensor-core rate measured under ideal conditions (and often quoted with the 2× sparsity bonus that real dense matmuls never see). In practice the accelerator waits on memory reads, on gradient all-reduces over the network, on pipeline bubbles, on kernels that are memory-bound, and on non-matmul work like softmax and layernorm. Well-tuned large runs land around 35–50% MFU; smaller models on big clusters do worse because communication and overhead dominate. MFU is measured, not assumed — which is why an honest estimate multiplies peak by it.

How many GPUs does it take to train a large language model?

It falls out of the 6ND rule and MFU. The number of GPUs is total compute ÷ (per-GPU peak × MFU × wall-clock seconds). Sizing a 70B model on 15T tokens: 6 × 70e9 × 15e12 ≈ 6.3e24 FLOPs. An H100 delivers about 990 dense BF16 TFLOPS; at 40% MFU that is roughly 396 effective TFLOPS per GPU. To finish in a month (about 2.6 million seconds) you need 6.3e24 ÷ (3.96e14 × 2.6e6) ≈ 6,100 H100s — and since real 70B-scale MFU runs nearer 28%, the honest figure is closer to 8,000–9,000. Meta's own Llama-3 70B run reports 6.4 million H100-hours, which is exactly this arithmetic in reverse.

GPU Arithmetic: the Numbers That Decide ML Systems

Three numbers decide every ML-infra design: FLOPs to compute, bytes to move, bytes to store — everything else derives. We read the A100/H100/H200 spec sheet honestly (the sparsity asterisk quietly doubles the headline), plot dialable workloads on a real machine's roofline, derive the matmul 2mnk and training 6ND rules once, reproduce Llama-3's GPU-hours, and end on a live estimator: params, tokens, GPUs, MFU, dtype → days and dollars. Drawn, computed, and animated.

Concept · AI / ML. The source ↗

A free, interactive, animated visual explainer of GPU Arithmetic: the Numbers That Decide ML Systems — built to be understood, not skimmed.

Questions

What are the three numbers that decide GPU workloads?: Every ML-infra estimate reduces to three quantities: the FLOPs a computation costs, the bytes it must move between memory and the compute units, and the bytes it must store. FLOPs set the compute floor, bytes-moved set the memory-bandwidth floor, and bytes-stored decide whether the thing fits on the GPUs you have. The actual run time of any kernel is the larger of (FLOPs ÷ peak compute) and (bytes ÷ memory bandwidth) — whichever wall you hit first. Almost every other figure you care about (throughput, cost, how many GPUs, how long) derives from these three, which is why estimating them well is the whole game.
What is arithmetic intensity and the roofline model?: Arithmetic intensity is the ratio of work to traffic — FLOPs performed per byte moved from memory (op/byte). The roofline model, from Williams, Waterman and Patterson (2009), plots attainable performance against arithmetic intensity on log-log axes: a slanted line (memory bandwidth × intensity) rising until it meets a flat ceiling (the chip's peak compute). Where they meet is the ridge point, equal to peak-FLOPs ÷ bandwidth — the machine balance. A workload whose intensity sits left of the ridge is memory-bound (starved for bandwidth, compute idle); right of the ridge it is compute-bound. Reading which side you land on tells you whether faster math or faster memory would help — and usually it is memory.
What is the 6ND rule for training FLOPs?: Training a dense transformer with N parameters on D tokens costs roughly 6ND floating-point operations. It comes from the matmul cost 2mnk: a forward pass touches every parameter once as a multiply-add per token, about 2N FLOPs per token; the backward pass computes gradients with respect to both the inputs and the weights, roughly twice the forward work, about 4N per token. Two plus four is six, so total training compute ≈ 6 × N × D. For Llama-3 8B on over 15 trillion tokens that is 6 × 8e9 × 15e12 ≈ 7.2e23 FLOPs — a single multiplication that lets you size a training run before you rent a single GPU.
What is MFU and why can't GPUs hit peak FLOPs?: MFU (Model FLOPs Utilization) is the fraction of a GPU's advertised peak compute that a real training run actually sustains — realized FLOPs ÷ peak FLOPs. Nobody hits 100%. The headline TFLOPS on a datasheet is a dense tensor-core rate measured under ideal conditions (and often quoted with the 2× sparsity bonus that real dense matmuls never see). In practice the accelerator waits on memory reads, on gradient all-reduces over the network, on pipeline bubbles, on kernels that are memory-bound, and on non-matmul work like softmax and layernorm. Well-tuned large runs land around 35–50% MFU; smaller models on big clusters do worse because communication and overhead dominate. MFU is measured, not assumed — which is why an honest estimate multiplies peak by it.
How many GPUs does it take to train a large language model?: It falls out of the 6ND rule and MFU. The number of GPUs is total compute ÷ (per-GPU peak × MFU × wall-clock seconds). Sizing a 70B model on 15T tokens: 6 × 70e9 × 15e12 ≈ 6.3e24 FLOPs. An H100 delivers about 990 dense BF16 TFLOPS; at 40% MFU that is roughly 396 effective TFLOPS per GPU. To finish in a month (about 2.6 million seconds) you need 6.3e24 ÷ (3.96e14 × 2.6e6) ≈ 6,100 H100s — and since real 70B-scale MFU runs nearer 28%, the honest figure is closer to 8,000–9,000. Meta's own Llama-3 70B run reports 6.4 million H100-hours, which is exactly this arithmetic in reverse.

GPU Arithmetic: the Numbers That Decide ML Systems

Questions

Related explainers