Inference & Serving

Turning a trained model into a service that answers thousands of streams at once: the memory-bandwidth roofline, the KV cache, continuous batching, paged attention, and the scheduler decisions that decide latency and cost per GPU-hour.

Explainers

• Quantization for Deployment — A trained model is a pile of fp16 numbers, and at inference the whole game is how many bytes you drag from memory per token — because decode is bandwidth-bound. Quantization shrinks each weight to int8 or int4: a scale, a zero-point, a rounding, and the error that leaks. We compute it live on a real weight row, show why activations break at int8, walk GPTQ and AWQ honestly, dial the size-vs-accuracy trade, and cover KV-cache quant, FP8/NVFP4, and ternary BitNet — drawn, computed, and animated.
• How Qwix Quantizes Any Flax Model Without Touching Its Code — You have a trained Flax model and you want it in int8 — but the model code isn't yours to edit. Qwix quantizes it anyway, by a trick that has nothing to do with the model: it swaps the ops the model calls. For the duration of one forward pass it patches jax.lax.dot_general and jnp.einsum, so every matmul routes through a quantized version while the model's own source never changes. We drive the op-swap live, compute int8/int4/fp8/nf4 grids from Qwix's real bounds, show how PTQ and QAT ride the exact same mechanism, and map what gets quantized where — plus the honest costs of patching a running program.
• GPTQ, Line by Line: Hessian-Based Weight Quantization — Round a weight to fewer bits and you commit an error — GPTQ's move is to make the weights you have not rounded yet absorb it. We walk Google's real JAX implementation (qwix's gptq_core.py, 220 lines) top to bottom: why the Hessian is just X·Xᵀ, the dampening that keeps a Cholesky alive, the three-line factorization dance that replaces a matrix inverse, and the two nested loops where each column's error becomes a rank-one update on everything to its right. Then the algorithm runs live — scrub the blocksize and damping and watch the compensation wave move.
• Design an LLM Serving Platform — One trained model, thousands of concurrent chat and batch streams, and a GPU that costs by the hour. Start from the roofline that rules everything — prefill is compute-bound, decode is memory-bandwidth-bound — then build up through the KV cache, continuous batching, paged attention, and prefill/decode disaggregation. Drawn, computed, and animated.
• Design a Text-to-Image Service — The image-generation sibling of the LLM serving platform — and it inverts almost everything. A chatbot streams tokens the reader consumes as they arrive; an image is all-or-nothing for whole seconds, so the product is a queue with a progress bar, not a stream. Build it up honestly: what a request really is (prompt → text encoder → N denoising steps → decode, with the diffusion loop as a black box), a commit-first estimate of seconds-and-cents per image, the three serving shapes deliberated interviewer-style, the GPU fleet alive under real queueing arithmetic, LoRA adapters for the personalized-headshot product, two-sided safety with an honest fail-open/fail-closed call, the failure sweep, and text-to-video as the same skeleton at brutal scale.