What is a hot key in a distributed cache and why is it a problem?

A hot key is a single key that receives a wildly disproportionate share of traffic — a celebrity post, a flash-sale item, a trending video. Because a key is deterministically mapped to one shard, all of that key’s reads land on one node while the rest of the fleet sits idle. A key pulling 300,000 requests a second can exceed a single node’s throughput budget on its own, so the cache is "balanced" on paper yet melting on one machine. The memcached paper names the write-side version of this a "thundering herd": "A thundering herd happens when a specific key undergoes heavy read and write activity."

How does consistent hashing reduce remapping when a cache node is added or removed?

With plain modulo sharding (shard = hash(key) mod N), changing N reshuffles nearly every key — going from 8 nodes to 7 remaps about 7/8 of the keyspace, and every remapped key is an instant miss that falls through to the database. Consistent hashing places nodes and keys on a ring and assigns each key to the next node clockwise, so adding or removing a node only moves the keys in that one node’s arc — roughly 1/N of the keyspace. Virtual nodes (many ring points per physical node) smooth the otherwise lumpy load distribution.

How do you handle a hot key that exceeds one node’s capacity?

Detect it, then spread it. Detection is cheap with a count-min sketch that estimates per-key request rates in fixed memory and surfaces the top-K heaviest keys. Absorption replicates the hot key onto several nodes and has clients fan out reads across the copies, so a 300k-QPS key split across four replicas is 75k per node. A per-app-server L1 cache with a jittered TTL absorbs most of the reads before they leave the box. The cost is fan-out on writes: every copy — and every L1 — must be invalidated on an update.

What is a cache stampede and how do you prevent it?

A cache stampede (thundering herd) happens when a hot key expires and every concurrent request misses at once, so all of them hit the database for the same value. At 100,000 requests a second with a 10 ms database read, that is about 1,000 duplicate queries in flight for one key. Single-flight (a lease) lets only the first miss recompute the value while the others wait and read the result it writes; probabilistic early recomputation refreshes the value slightly before expiry, with rising probability as the TTL nears, so a hot key is never all-expired under load.

Why delete the cache entry on a write instead of updating it in place?

Because delete is idempotent and update is not. Under concurrent writers, two in-place updates can be applied to the cache out of order and leave a stale value cached forever; a delete just removes the entry, and the next reader repopulates it from the database on demand. This is the cache-aside (look-aside) pattern: read through the cache, on a miss load from the database and populate, and on a write update the database and delete the cached key. There is still a small read-after-write race window, which single-flight and short TTLs bound.

Design a Distributed Cache

Start with one box in front of the database; end with a sharded, replicated cache that survives a celebrity key pulling 300k requests a second. Consistent hashing, cache-aside, eviction, and the hot-key problem — detection, absorption, and stampede protection — drawn and computed.

System design · Systems. The source ↗

A free, interactive, animated visual explainer of Design a Distributed Cache — built to be understood, not skimmed.

Questions

What is a hot key in a distributed cache and why is it a problem?: A hot key is a single key that receives a wildly disproportionate share of traffic — a celebrity post, a flash-sale item, a trending video. Because a key is deterministically mapped to one shard, all of that key’s reads land on one node while the rest of the fleet sits idle. A key pulling 300,000 requests a second can exceed a single node’s throughput budget on its own, so the cache is "balanced" on paper yet melting on one machine. The memcached paper names the write-side version of this a "thundering herd": "A thundering herd happens when a specific key undergoes heavy read and write activity."
How does consistent hashing reduce remapping when a cache node is added or removed?: With plain modulo sharding (shard = hash(key) mod N), changing N reshuffles nearly every key — going from 8 nodes to 7 remaps about 7/8 of the keyspace, and every remapped key is an instant miss that falls through to the database. Consistent hashing places nodes and keys on a ring and assigns each key to the next node clockwise, so adding or removing a node only moves the keys in that one node’s arc — roughly 1/N of the keyspace. Virtual nodes (many ring points per physical node) smooth the otherwise lumpy load distribution.
How do you handle a hot key that exceeds one node’s capacity?: Detect it, then spread it. Detection is cheap with a count-min sketch that estimates per-key request rates in fixed memory and surfaces the top-K heaviest keys. Absorption replicates the hot key onto several nodes and has clients fan out reads across the copies, so a 300k-QPS key split across four replicas is 75k per node. A per-app-server L1 cache with a jittered TTL absorbs most of the reads before they leave the box. The cost is fan-out on writes: every copy — and every L1 — must be invalidated on an update.
What is a cache stampede and how do you prevent it?: A cache stampede (thundering herd) happens when a hot key expires and every concurrent request misses at once, so all of them hit the database for the same value. At 100,000 requests a second with a 10 ms database read, that is about 1,000 duplicate queries in flight for one key. Single-flight (a lease) lets only the first miss recompute the value while the others wait and read the result it writes; probabilistic early recomputation refreshes the value slightly before expiry, with rising probability as the TTL nears, so a hot key is never all-expired under load.
Why delete the cache entry on a write instead of updating it in place?: Because delete is idempotent and update is not. Under concurrent writers, two in-place updates can be applied to the cache out of order and leave a stale value cached forever; a delete just removes the entry, and the next reader repopulates it from the database on demand. This is the cache-aside (look-aside) pattern: read through the cache, on a miss load from the database and populate, and on a write update the database and delete the cached key. There is still a small read-after-write race window, which single-flight and short TTLs bound.

Design a Distributed Cache

Questions

Related explainers