How does a time-series database store metrics so cheaply?

It never stores a metric as a row. Samples for one series are appended to a write-ahead log, then packed into immutable blocks where timestamps are delta-of-delta encoded and float values are XOR-encoded against the previous value — Facebook’s Gorilla scheme. Because scrapes land at regular intervals, "about 96% of all time stamps can be compressed to a single bit," and Gorilla compresses a 16-byte (timestamp, value) pair "down to an average of 1.37 bytes, a 12x reduction in size." Prometheus, built on the same ideas, "stores an average of only 1-2 bytes per sample."

What is cardinality and why is it the enemy in monitoring?

A time series is uniquely identified by a metric name plus its full set of label key-value pairs, so the number of active series is the product of every label’s cardinality. Attach a high-cardinality label — a user ID, a request ID, an email — and the series count multiplies catastrophically: a metric already at hundreds of thousands of series times a 10,000-value customer_id label is billions of series, each needing its own in-memory chunk and index entry. Active series, not sample volume, is what sets the memory ceiling, which is why ingesters enforce a hard series limit and reject new series past it.

Pull or push for collecting metrics?

Pull (the server scrapes targets on a schedule, as Prometheus does) gets liveness monitoring for free — a scrape that fails is itself a signal the target is down — and lets the server control its own ingest rate. Push (targets send to a gateway, as StatsD/Graphite do) fits short-lived and batch jobs that vanish before any scrape, and firewalled targets that can’t be reached inbound. Real systems do both: pull for long-running services, a push gateway for the ephemeral edge.

How do you serve a one-year range query without scanning a year of raw data?

Downsampling. Older blocks are rewritten at coarser resolutions — Thanos keeps raw, 5-minute, and 1-hour tiers — and the query layer routes each query to the coarsest resolution that still answers it: a 1-hour dashboard reads raw samples, a 1-year dashboard reads the 1-hour rollup. Thanos is explicit that downsampling is "not saving disk space" — it adds blocks — its purpose is "to get fast results for range queries of big time intervals like months or years."

What monitors the monitoring system when it fails during an outage?

A dead man’s switch. You cannot use the metrics system to detect that the metrics system is down, so you invert the logic: define one alert rule that is always firing and route it to an external heartbeat receiver. As long as the pipeline is healthy the heartbeat arrives on schedule; when the pipeline dies the heartbeat stops, and the external monitor pages precisely because it heard silence. The alert firing is normal; its absence is the signal.

Design a Metrics & Monitoring System

From a metrics table in Postgres to a purpose-built time-series database — the append-only TSDB, delta-of-delta plus XOR compression, the cardinality bomb that decides whether it fits in RAM, downsampling, and an alerting pipeline that watches itself.

System design · Systems. The source ↗

A free, interactive, animated visual explainer of Design a Metrics & Monitoring System — built to be understood, not skimmed.

Questions

How does a time-series database store metrics so cheaply?: It never stores a metric as a row. Samples for one series are appended to a write-ahead log, then packed into immutable blocks where timestamps are delta-of-delta encoded and float values are XOR-encoded against the previous value — Facebook’s Gorilla scheme. Because scrapes land at regular intervals, "about 96% of all time stamps can be compressed to a single bit," and Gorilla compresses a 16-byte (timestamp, value) pair "down to an average of 1.37 bytes, a 12x reduction in size." Prometheus, built on the same ideas, "stores an average of only 1-2 bytes per sample."
What is cardinality and why is it the enemy in monitoring?: A time series is uniquely identified by a metric name plus its full set of label key-value pairs, so the number of active series is the product of every label’s cardinality. Attach a high-cardinality label — a user ID, a request ID, an email — and the series count multiplies catastrophically: a metric already at hundreds of thousands of series times a 10,000-value customer_id label is billions of series, each needing its own in-memory chunk and index entry. Active series, not sample volume, is what sets the memory ceiling, which is why ingesters enforce a hard series limit and reject new series past it.
Pull or push for collecting metrics?: Pull (the server scrapes targets on a schedule, as Prometheus does) gets liveness monitoring for free — a scrape that fails is itself a signal the target is down — and lets the server control its own ingest rate. Push (targets send to a gateway, as StatsD/Graphite do) fits short-lived and batch jobs that vanish before any scrape, and firewalled targets that can’t be reached inbound. Real systems do both: pull for long-running services, a push gateway for the ephemeral edge.
How do you serve a one-year range query without scanning a year of raw data?: Downsampling. Older blocks are rewritten at coarser resolutions — Thanos keeps raw, 5-minute, and 1-hour tiers — and the query layer routes each query to the coarsest resolution that still answers it: a 1-hour dashboard reads raw samples, a 1-year dashboard reads the 1-hour rollup. Thanos is explicit that downsampling is "not saving disk space" — it adds blocks — its purpose is "to get fast results for range queries of big time intervals like months or years."
What monitors the monitoring system when it fails during an outage?: A dead man’s switch. You cannot use the metrics system to detect that the metrics system is down, so you invert the logic: define one alert rule that is always firing and route it to an external heartbeat receiver. As long as the pipeline is healthy the heartbeat arrives on schedule; when the pipeline dies the heartbeat stops, and the external monitor pages precisely because it heard silence. The alert firing is normal; its absence is the signal.

Design a Metrics & Monitoring System

Questions

Related explainers