Inference & Serving

Turning a trained model into a service that answers thousands of streams at once: the memory-bandwidth roofline, the KV cache, continuous batching, paged attention, and the scheduler decisions that decide latency and cost per GPU-hour.

Explainers

• Design an LLM Serving Platform — One trained model, thousands of concurrent chat and batch streams, and a GPU that costs by the hour. Start from the roofline that rules everything — prefill is compute-bound, decode is memory-bandwidth-bound — then build up through the KV cache, continuous batching, paged attention, and prefill/decode disaggregation. Drawn, computed, and animated.