Streaming

Computing over unbounded data as it arrives: stateful operators, event time and late data, consistent snapshots, and exactly-once through crashes.

Explainers

• Top-K & Heavy Hitters — Who are the top ten right now? Easy at a thousand rows, brutal at a billion. Two questions wear one name: a live leaderboard of 25M players, and the heavy hitters of a firehose you can never store. Built from zero — why SQL ORDER BY dies, how a Redis sorted set answers rank in O(log n) with a skip list, how the board shards and why hashing breaks the rank query, then the streaming half: a count-min sketch that counts a billion keys in a kilobyte (and sometimes lies), a min-heap of the top ten, and lambda reconciliation when the number has to be billing-exact.
• Design a Stream Processor with Exactly-Once — How a system that counts events as they stream past can crash, recover, and never count anything twice: periodic snapshots of its memory, an honest way to handle events that arrive late, and an output stage that only commits when everything agrees — drawn, computed, and animated.
• Design Ad-Click Aggregation (Lambda vs Kappa) — Count a billion ad clicks a day two ways at once — a dashboard fresh within seconds, and a total accurate enough to bill against. How duplicate clicks are weeded out, how late-arriving clicks are handled honestly, and why one replayable pipeline can do the work of two.