Compilers & Kernels

What happens between your model code and the silicon: the compiler that splits one program across a device mesh, decides which operations fuse, assigns every buffer an address, and emits the collectives that move tensors between chips.

Explainers

• The SPMD Partitioner: One Program Across a Device Mesh — You write a program as if one giant machine ran it, annotate a few tensors with how they are split across a mesh of chips, and the compiler rewrites it into N identical per-device programs — inserting every all-gather, reduce-scatter, all-to-all, and halo exchange needed to keep the maths correct. We trace how those annotations propagate through the graph, where each collective is born (and which resharding forces it), the one windowed op that needs halo exchange, and the case the partitioner cannot solve on its own — drawn, computed, and animated.