What is quantization in LLMs?

Quantization stores each model weight (and sometimes each activation) in fewer bits — int8 or int4 instead of fp16. You map a group of floating-point numbers onto a small grid of integers with an affine rule: pick a scale (how much real value each integer step is worth) and a zero-point (which integer maps to zero), then round every weight to the nearest grid point. Dequantization multiplies back: x ≈ scale × (q − zero_point). The gap between the original value and its reconstruction is the quantization error, and the whole craft is keeping that error small where it matters.

Why does quantization make inference faster?

Because LLM decode — generating one token at a time — is memory-bandwidth-bound, not compute-bound. To produce a single token the GPU streams every weight in the model from memory, does a little math, and moves on; the bottleneck is bytes moved, not FLOPs. Halve the bytes per weight (fp16 → int8) and you roughly halve the memory traffic per token, so decode throughput rises close to the byte ratio. The speedup is sublinear in practice — the KV cache, activations, and dequantization overhead do not shrink by the same factor — but the direction is real and large.

What is the difference between GPTQ and AWQ?

Both are post-training weight-only quantization methods that push to 3–4 bits with small accuracy loss, but they attack the error differently. GPTQ quantizes weights one at a time using approximate second-order (Hessian) information, and after rounding each weight it adjusts the remaining un-quantized weights to compensate for the error just introduced — error-compensating rounding. AWQ instead observes that a tiny fraction of weight channels are salient (identified by activation magnitude, not weight magnitude) and protects them by scaling those channels up before quantizing, which shrinks their relative error. GPTQ compensates after the fact; AWQ protects the important channels up front.

What is KV-cache quantization?

During generation, a model caches the key and value vectors for every past token so it does not recompute attention from scratch — the KV cache. For long contexts and large batches this cache can rival or exceed the weights in memory, and since decode is bandwidth-bound, reading it dominates latency. Quantizing the KV cache to int8 or int4 halves or quarters those bytes, which both frees memory (longer contexts, bigger batches) and speeds decode. It is a second, independent win on top of weight quantization, and it is often more delicate because keys and values carry outliers of their own.

When should you not quantize a model?

When the accuracy you lose costs more than the memory or speed you gain. If the model already fits comfortably and serves fast enough, aggressive quantization only adds risk. Very low bit-widths (int3 and below) can degrade reasoning, math, and long-context tasks in ways that generic perplexity numbers hide, so a task-specific evaluation is mandatory before shipping. Quantization is also not free to run — dequantization kernels and mixed-precision paths need real hardware and library support — and if your bottleneck is compute (large-batch prefill) rather than memory bandwidth, weight quantization may buy far less than expected.

Quantization for Deployment

A trained model is a pile of fp16 numbers, and at inference the whole game is how many bytes you drag from memory per token — because decode is bandwidth-bound. Quantization shrinks each weight to int8 or int4: a scale, a zero-point, a rounding, and the error that leaks. We compute it live on a real weight row, show why activations break at int8, walk GPTQ and AWQ honestly, dial the size-vs-accuracy trade, and cover KV-cache quant, FP8/NVFP4, and ternary BitNet — drawn, computed, and animated.

Concept · AI / ML. The source ↗

A free, interactive, animated visual explainer of Quantization for Deployment — built to be understood, not skimmed.

Questions

What is quantization in LLMs?: Quantization stores each model weight (and sometimes each activation) in fewer bits — int8 or int4 instead of fp16. You map a group of floating-point numbers onto a small grid of integers with an affine rule: pick a scale (how much real value each integer step is worth) and a zero-point (which integer maps to zero), then round every weight to the nearest grid point. Dequantization multiplies back: x ≈ scale × (q − zero_point). The gap between the original value and its reconstruction is the quantization error, and the whole craft is keeping that error small where it matters.
Why does quantization make inference faster?: Because LLM decode — generating one token at a time — is memory-bandwidth-bound, not compute-bound. To produce a single token the GPU streams every weight in the model from memory, does a little math, and moves on; the bottleneck is bytes moved, not FLOPs. Halve the bytes per weight (fp16 → int8) and you roughly halve the memory traffic per token, so decode throughput rises close to the byte ratio. The speedup is sublinear in practice — the KV cache, activations, and dequantization overhead do not shrink by the same factor — but the direction is real and large.
What is the difference between GPTQ and AWQ?: Both are post-training weight-only quantization methods that push to 3–4 bits with small accuracy loss, but they attack the error differently. GPTQ quantizes weights one at a time using approximate second-order (Hessian) information, and after rounding each weight it adjusts the remaining un-quantized weights to compensate for the error just introduced — error-compensating rounding. AWQ instead observes that a tiny fraction of weight channels are salient (identified by activation magnitude, not weight magnitude) and protects them by scaling those channels up before quantizing, which shrinks their relative error. GPTQ compensates after the fact; AWQ protects the important channels up front.
What is KV-cache quantization?: During generation, a model caches the key and value vectors for every past token so it does not recompute attention from scratch — the KV cache. For long contexts and large batches this cache can rival or exceed the weights in memory, and since decode is bandwidth-bound, reading it dominates latency. Quantizing the KV cache to int8 or int4 halves or quarters those bytes, which both frees memory (longer contexts, bigger batches) and speeds decode. It is a second, independent win on top of weight quantization, and it is often more delicate because keys and values carry outliers of their own.
When should you not quantize a model?: When the accuracy you lose costs more than the memory or speed you gain. If the model already fits comfortably and serves fast enough, aggressive quantization only adds risk. Very low bit-widths (int3 and below) can degrade reasoning, math, and long-context tasks in ways that generic perplexity numbers hide, so a task-specific evaluation is mandatory before shipping. Quantization is also not free to run — dequantization kernels and mixed-precision paths need real hardware and library support — and if your bottleneck is compute (large-batch prefill) rather than memory bandwidth, weight quantization may buy far less than expected.

Quantization for Deployment

Questions

Related explainers