What are neural scaling laws?

Scaling laws are empirical power-law relationships that predict a language model's test loss from three inputs: the number of parameters, the number of training tokens, and the total compute spent. Kaplan et al. (2020) found that "the loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude." On a log-log plot each is a straight line, so loss is predictable across many orders of magnitude — which lets you allocate a fixed compute budget rather than guess a model size.

What is the Chinchilla scaling law and the 20 tokens per parameter rule?

Chinchilla (Hoffmann et al., 2022) found that for compute-optimal training, parameters and training tokens should grow together — "for every doubling of model size the number of training tokens should also be doubled." In practice the compute-optimal frontier sits near 20 training tokens per parameter: a 70B model wants roughly 1.4 trillion tokens. The paper trained a 70B model, Chinchilla, on 4× more data than the 280B Gopher at the same compute, and it "uniformly and significantly outperforms Gopher (280B), GPT-3 (175B)" and others.

Why did the compute-optimal recommendation change from Kaplan to Chinchilla?

Kaplan's 2020 analysis concluded that optimal model size grows much faster than data — N ∝ C^0.73, so a 10× compute increase meant about 5.5× the parameters but only 1.8× the tokens. That drove the parameter race (GPT-3, Gopher, Megatron-Turing). Chinchilla re-ran the experiment more carefully and found both exponents near 0.5, so a 10× budget means about 3.2× parameters and 3.2× tokens. A 2024 replication traced Kaplan's overestimate largely to counting non-embedding parameters and fitting at smaller scale. The upshot: the models of that era, GPT-3 included, were significantly undertrained.

Why do modern models like Llama 3 train far past Chinchilla-optimal?

Because Chinchilla only minimizes training compute, and it ignores inference. Once a model serves billions of requests, a smaller model trained on far more data is cheaper for its whole life even if training cost more, because every inference is cheaper. Sardana et al. (2023) show that "LLM researchers expecting reasonably large inference demand (~1B requests) should train models smaller and longer than Chinchilla-optimal." Llama 3 8B trains on about 15 trillion tokens — roughly 1,900 tokens per parameter, nearly 100× the Chinchilla ratio — a deliberate over-train to make inference cheap.

Are emergent abilities of large language models real or a mirage?

It is contested, and the honest answer is "partly a measurement artifact." Wei et al. (2022) defined an ability as emergent if "it is not present in smaller models but is present in larger models" and cannot be predicted by extrapolation. Schaeffer et al. (2023) pushed back: "emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale" — under a continuous metric the same curves are smooth and predictable. Sharp, all-or-nothing scoring (exact-match on a long answer) manufactures apparent jumps; the underlying loss still moves along its power law.

Scaling Laws, Honestly

Language-model loss falls as a clean power law in compute — which turns "how big a model?" from taste into arithmetic. Follow Kaplan's parameter race into Chinchilla's correction (tokens scale WITH parameters, about 20 per one), dial a FLOPs budget to its compute-optimal split, plot GPT-3, Chinchilla and Llama 3 honestly on the line, and see why inference economics now makes everyone over-train — with an honest beat on whether "emergent abilities" are real.

Concept · AI / ML. The source ↗

A free, interactive, animated visual explainer of Scaling Laws, Honestly — built to be understood, not skimmed.

Questions

What are neural scaling laws?: Scaling laws are empirical power-law relationships that predict a language model's test loss from three inputs: the number of parameters, the number of training tokens, and the total compute spent. Kaplan et al. (2020) found that "the loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude." On a log-log plot each is a straight line, so loss is predictable across many orders of magnitude — which lets you allocate a fixed compute budget rather than guess a model size.
What is the Chinchilla scaling law and the 20 tokens per parameter rule?: Chinchilla (Hoffmann et al., 2022) found that for compute-optimal training, parameters and training tokens should grow together — "for every doubling of model size the number of training tokens should also be doubled." In practice the compute-optimal frontier sits near 20 training tokens per parameter: a 70B model wants roughly 1.4 trillion tokens. The paper trained a 70B model, Chinchilla, on 4× more data than the 280B Gopher at the same compute, and it "uniformly and significantly outperforms Gopher (280B), GPT-3 (175B)" and others.
Why did the compute-optimal recommendation change from Kaplan to Chinchilla?: Kaplan's 2020 analysis concluded that optimal model size grows much faster than data — N ∝ C^0.73, so a 10× compute increase meant about 5.5× the parameters but only 1.8× the tokens. That drove the parameter race (GPT-3, Gopher, Megatron-Turing). Chinchilla re-ran the experiment more carefully and found both exponents near 0.5, so a 10× budget means about 3.2× parameters and 3.2× tokens. A 2024 replication traced Kaplan's overestimate largely to counting non-embedding parameters and fitting at smaller scale. The upshot: the models of that era, GPT-3 included, were significantly undertrained.
Why do modern models like Llama 3 train far past Chinchilla-optimal?: Because Chinchilla only minimizes training compute, and it ignores inference. Once a model serves billions of requests, a smaller model trained on far more data is cheaper for its whole life even if training cost more, because every inference is cheaper. Sardana et al. (2023) show that "LLM researchers expecting reasonably large inference demand (~1B requests) should train models smaller and longer than Chinchilla-optimal." Llama 3 8B trains on about 15 trillion tokens — roughly 1,900 tokens per parameter, nearly 100× the Chinchilla ratio — a deliberate over-train to make inference cheap.
Are emergent abilities of large language models real or a mirage?: It is contested, and the honest answer is "partly a measurement artifact." Wei et al. (2022) defined an ability as emergent if "it is not present in smaller models but is present in larger models" and cannot be predicted by extrapolation. Schaeffer et al. (2023) pushed back: "emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale" — under a continuous metric the same curves are smooth and predictable. Sharp, all-or-nothing scoring (exact-match on a long answer) manufactures apparent jumps; the underlying loss still moves along its power law.

Scaling Laws, Honestly

Questions

Related explainers