How do you send a report to millions of customers on a deadline?

Split the work into two systems with a durable seam between them. First, a periodic batch pipeline computes each customer’s report as a DAG of aggregations pinned to one point-in-time snapshot of the data, renders it into an immutable artifact (~500 KB), and stores it in object storage. Second, a separate delivery ramp reads an ordered worklist and mails everyone at a metered pace. The split matters because compute and delivery have opposite tempos — an hour of parallel batch compute versus nearly three hours of rate-limited sending — and welding them (the naive cron loop) means a slow report stalls the sends and a provider hiccup stalls the compute. With 20 million reports and a 3-hour window you need to sustain about 1,900 sends per second, which is why every second of send capacity has to be spent well.

Should you precompute every report or render each one on open?

Usually neither extreme — a hybrid, tuned to the open rate. Precomputing all 20 million buys an instant open (the link resolves straight to a finished artifact) but wastes compute and storage on the majority of reports nobody ever opens; at a 35% open rate you’ve rendered and stored 13 million reports for nothing. Rendering on open wastes nothing but gives a slow first click and a thundering herd the moment the emails land and everyone opens at once. The senior answer precomputes the segment most likely to be opened — paying tiers, active users — and renders the long tail lazily. Set the precomputed fraction near the actual open rate and you get most of the instant experience at a fraction of the waste; getting the prediction wrong only costs money, never correctness.

How do you avoid tripping the email provider’s rate limit?

Shape the send rate with a token bucket set to the provider’s ceiling. Providers like Amazon SES enforce a maximum sending rate — SES states you “can exceed this quota for short bursts, but not for sustained periods of time,” which is exactly token-bucket behavior: a token per send, refilled at the sustained ceiling, with a small burst reserve. Point every delivery worker at one shared token bucket (a single counter in a fast store like Redis, not per-worker) and the whole fleet is physically incapable of exceeding the sustained cap no matter how many workers you run. Shape to slightly below the stated cap, since providers throttle probabilistically near the limit and the accepted rate “can be less than the maximum.”

How do you make sure a customer never gets the report twice?

With idempotency keys plus a checkpointed cursor. Over a real network a send times out and you can’t tell whether it went through, so you must retry — safely. Each send carries a stable idempotency key (here, run-id / customer-id); a dedupe store or the provider recognizes a repeat and returns the original result instead of sending again. Delivery is still at-least-once, but the effect is exactly-once because the duplicate is dropped. On top of that, progress lives in a committed cursor into the ordered worklist rather than in a worker’s memory, so a crash resumes from where it left off instead of restarting and re-mailing everyone already done. Commit the cursor after the send and lean on the key to make the ambiguous re-send harmless.

How is this different from a notification system?

A notification system is event-driven: it reacts to a stream of individual events (“your driver is arriving”) and routes each to a phone across push, SMS, and email in real time. This is a scheduled batch: one enormous, bounded worklist, computed ahead of time from a data snapshot, drained on a deadline. The delivery mechanics rhyme — queues, retries, rate caps, idempotency, suppression — but the shape is a fixed list with a finish line, not an open-ended firehose, and the interesting half here is the report compute (a DAG, the precompute deliberation, immutable artifacts behind signed links) that a notification system doesn’t have. They’re sibling designs; this page links the notification system rather than duplicating it.

Build a Report and Fan It Out to Millions

One prompt hiding two systems: a periodic batch pipeline that computes 20 million personalized reports, joined to a delivery ramp that mails every one inside a 3-hour window over a provider you don’t own. Built from zero: the scope split (compute vs deliver), a commit-first envelope where a ~2,000/s send ceiling barely clears the deadline, the naive cron loop and why it dies, the compute DAG with a pinned snapshot and per-task retries, the central precompute-vs-render-on-open deliberation tuned to the open rate, rendering to immutable artifacts behind short-lived signed links, then the delivery half — batching, a token-bucket rate shaper clamped to the provider ceiling, idempotency keys so nobody gets the report twice, suppression before the send, a checkpointed cursor so a crash resumes instead of restarting, and a live completion burn-down that proves all N went out by when. Distinct from a notification system (linked, not duplicated): this is the report COMPUTE joined to the delivery ramp.

System design · Systems. The source ↗

A free, interactive, animated visual explainer of Build a Report and Fan It Out to Millions — built to be understood, not skimmed.

Questions

How do you send a report to millions of customers on a deadline?: Split the work into two systems with a durable seam between them. First, a periodic batch pipeline computes each customer’s report as a DAG of aggregations pinned to one point-in-time snapshot of the data, renders it into an immutable artifact (~500 KB), and stores it in object storage. Second, a separate delivery ramp reads an ordered worklist and mails everyone at a metered pace. The split matters because compute and delivery have opposite tempos — an hour of parallel batch compute versus nearly three hours of rate-limited sending — and welding them (the naive cron loop) means a slow report stalls the sends and a provider hiccup stalls the compute. With 20 million reports and a 3-hour window you need to sustain about 1,900 sends per second, which is why every second of send capacity has to be spent well.
Should you precompute every report or render each one on open?: Usually neither extreme — a hybrid, tuned to the open rate. Precomputing all 20 million buys an instant open (the link resolves straight to a finished artifact) but wastes compute and storage on the majority of reports nobody ever opens; at a 35% open rate you’ve rendered and stored 13 million reports for nothing. Rendering on open wastes nothing but gives a slow first click and a thundering herd the moment the emails land and everyone opens at once. The senior answer precomputes the segment most likely to be opened — paying tiers, active users — and renders the long tail lazily. Set the precomputed fraction near the actual open rate and you get most of the instant experience at a fraction of the waste; getting the prediction wrong only costs money, never correctness.
How do you avoid tripping the email provider’s rate limit?: Shape the send rate with a token bucket set to the provider’s ceiling. Providers like Amazon SES enforce a maximum sending rate — SES states you “can exceed this quota for short bursts, but not for sustained periods of time,” which is exactly token-bucket behavior: a token per send, refilled at the sustained ceiling, with a small burst reserve. Point every delivery worker at one shared token bucket (a single counter in a fast store like Redis, not per-worker) and the whole fleet is physically incapable of exceeding the sustained cap no matter how many workers you run. Shape to slightly below the stated cap, since providers throttle probabilistically near the limit and the accepted rate “can be less than the maximum.”
How do you make sure a customer never gets the report twice?: With idempotency keys plus a checkpointed cursor. Over a real network a send times out and you can’t tell whether it went through, so you must retry — safely. Each send carries a stable idempotency key (here, run-id / customer-id); a dedupe store or the provider recognizes a repeat and returns the original result instead of sending again. Delivery is still at-least-once, but the effect is exactly-once because the duplicate is dropped. On top of that, progress lives in a committed cursor into the ordered worklist rather than in a worker’s memory, so a crash resumes from where it left off instead of restarting and re-mailing everyone already done. Commit the cursor after the send and lean on the key to make the ambiguous re-send harmless.
How is this different from a notification system?: A notification system is event-driven: it reacts to a stream of individual events (“your driver is arriving”) and routes each to a phone across push, SMS, and email in real time. This is a scheduled batch: one enormous, bounded worklist, computed ahead of time from a data snapshot, drained on a deadline. The delivery mechanics rhyme — queues, retries, rate caps, idempotency, suppression — but the shape is a fixed list with a finish line, not an open-ended firehose, and the interesting half here is the report compute (a DAG, the precompute deliberation, immutable artifacts behind signed links) that a notification system doesn’t have. They’re sibling designs; this page links the notification system rather than duplicating it.

Build a Report and Fan It Out to Millions

Questions

Related explainers