Fundamentals

The architecture under every modern model, built from first principles: tokens and embeddings, attention computed number by number, multi-head and the KV cache, positional encoding, and the block stacked into a full network.

Explainers

• Evals & Experimental Design — Most ML claims die at one question: compared to what? This is the research-taste page — the difference between "my method gets 71.2%" and "my method is actually better." We walk the claim-evidence gap, the tuned-baseline rule, the eval hierarchy from loss to production, and benchmark contamination — then the part everyone skips: is 71.2% really better than 70.8%? Dial an interactive significance explorer with real binomial confidence intervals and a live bootstrap, watch the variance a single model has run-to-run, judge the LLM judges honestly, and design the experiment before you run it.
• Optimization Dynamics: Why Adam, Why Warmup, Why Cosine — An optimizer never sees the loss surface — it gets one gradient at a time and has to turn that stream of local hints into a path to the bottom. We race SGD, momentum, and Adam down the same dialable ill-conditioned valley (real update rules, trajectories drawn live), then build Adam precisely — per-parameter learning rates from the second moment, bias correction honestly — and AdamW's decoupled decay. Then the practitioner's layer: why warmup exists, what cosine/linear/WSD actually buy, gradient clipping and loss spikes, muP transfer, and batch-size-vs-LR — all computed, all animated.
• Mixture of Experts, Routed Honestly — A dense model spends every parameter on every token; a mixture-of-experts model keeps far more parameters but touches only a slice per token — huge in memory, cheap in compute at once. The whole design turns on one question: which experts does a token go to? We compute the routing decision live — a real softmax gate over 8 experts, top-k selection, a capacity cap that drops overflow tokens, and the load-balancing loss that fights router collapse — then count Mixtral 8x7B down from 47B parameters to the 13B it actually uses, cover expert parallelism and its all-to-all bill, and the DeepSeek fine-grained-plus-shared recipe.
• GPU Arithmetic: the Numbers That Decide ML Systems — Three numbers decide every ML-infra design: FLOPs to compute, bytes to move, bytes to store — everything else derives. We read the A100/H100/H200 spec sheet honestly (the sparsity asterisk quietly doubles the headline), plot dialable workloads on a real machine's roofline, derive the matmul 2mnk and training 6ND rules once, reproduce Llama-3's GPU-hours, and end on a live estimator: params, tokens, GPUs, MFU, dtype → days and dollars. Drawn, computed, and animated.
• Scaling Laws, Honestly — Language-model loss falls as a clean power law in compute — which turns "how big a model?" from taste into arithmetic. Follow Kaplan's parameter race into Chinchilla's correction (tokens scale WITH parameters, about 20 per one), dial a FLOPs budget to its compute-optimal split, plot GPT-3, Chinchilla and Llama 3 honestly on the line, and see why inference economics now makes everyone over-train — with an honest beat on whether "emergent abilities" are real.
• Diffusion Models, Honestly — You can't paint a photo in one stroke, but you can clean up a slightly-noisy one — and if you can do that, you can generate. That is the whole trick. We destroy a real 2-D distribution on the DDPM noise schedule (the ᾱ_t math, computed live as you scrub time), state the ε-prediction objective in one honest equation, count the network evaluations a 50-step image really costs, work the arithmetic that makes latent diffusion 48× cheaper than pixels, dial classifier-free guidance between fidelity and diversity, and end on the honest history — why the field left GANs behind — and the failure modes nobody screenshots.
• The Transformer, End to End — The researcher-round staple: draw the architecture from memory and explain every tensor. We build it once, honestly — token → embedding → attention computed on four real tokens (Q/K/V, a QKᵀ heatmap, softmax, the weighted sum, every number real) → multi-head → the KV cache and the MHA/MQA/GQA/MLA memory ladder dialed live → RoPE as a rotation → the MLP and residual highway → norm placement → the block stacked N times → a parameter count that reproduces Llama-3-8B to the last billion.