How does HMAC webhook signature verification work, and why does the timestamp matter?

The sender computes an HMAC-SHA256 digest over a string built from the request, using a secret only the sender and receiver know, and attaches it as a header; the receiver recomputes the same digest and checks it matches. Stripe's own docs give the exact recipe: the signed string is "the timestamp (as a string), the character '.', the actual JSON payload," hashed with "the endpoint's signing secret as the key." The timestamp is what stops a replay: without it, a captured, legitimately-signed request could be resent forever with a still-valid signature. Stripe checks the timestamp against "a default tolerance of 5 minutes between the timestamp and the current time" and warns that "using a tolerance value of 0 disables the recency check entirely." Both Stripe and GitHub also insist the two signatures be compared with a constant-time function, never a plain ==, because an early-exit string comparison leaks how many leading bytes matched through response timing.

What retry schedule do production webhook platforms actually use, and why is jitter necessary?

They all use exponential backoff — retries spaced further apart the longer something stays broken — but the exact shape differs. Svix publishes its ladder outright: "Immediately, 5 seconds, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours, and 10 hours," eight attempts summing to about 27.6 hours from first failure to last. Stripe instead commits to a duration: it "attempts to deliver events to your destination for up to three days with an exponential backoff in live mode." Jitter — randomizing each retry's timing around its nominal gap — matters because without it, every endpoint that failed at the same moment (a shared load-balancer blip, say) retries in lockstep: all their 5-second retries land in the same instant, all their 5-minute retries land in the same instant, and so on — a synchronized "thundering herd" hitting infrastructure that just came back online and hasn't warmed up.

Why can't webhook delivery be exactly-once, and what should a receiver actually do about it?

Because the sender can never fully distinguish "the endpoint failed to process this" from "the endpoint processed it but the success response got lost in transit" — from the sender's side those look identical, so the only safe move is to retry, which means the same event can arrive twice. Stripe states the honest contract directly: "webhook endpoints might occasionally receive the same event more than once. You can guard against duplicated event receipts by logging the event IDs you've processed, and then not processing already-logged events." That is at-least-once delivery: never silently lost, occasionally duplicated. The fix lives entirely on the receiving end — give every event a stable unique ID and make processing idempotent (applying the same event twice has the same effect as applying it once), typically by checking that ID against a log of what's already been handled before acting on it again.

How do you stop one dead customer endpoint from degrading delivery to everyone else?

By making isolation structural rather than a shared resource everyone hopes not to abuse: give every registered endpoint its own queue, or a per-endpoint token bucket — a small allowance of sends that refills steadily and is spent one token per request. When one endpoint's bucket empties (because it's down or its retries are piling up), only that endpoint's deliveries wait; every other endpoint's bucket keeps refilling and keeps sending, untouched. This is what actually answers the "ordering vs. throughput" tension too: a customer who demands strict per-endpoint ordering forces serialization, but only within their own isolated queue — their slow server costs them latency, never the other thousands of endpoints sharing the platform. Left unfixed — one shared queue for every destination, delivered strictly in order — a single dead endpoint's retries create head-of-line blocking that delays healthy endpoints stuck behind it in the same queue, which is the naive design's failure mode reappearing one layer down.

Design a Webhook & Trigger Delivery Platform

Thousands of customer servers, each unpredictable — slow, offline, flaky at 3am — and one bad one must never touch the rest. We compute the fan-out arithmetic (events → deliveries, retry amplification when 5% of endpoints are down, storage for a 3-day retry horizon), watch a naive POST-in-the-request-loop stall on a single slow endpoint, then build the real design one mechanism at a time: a durable queue off the request path, HMAC signature verification in both directions, per-endpoint token buckets, a jittered retry ladder to a dead-letter queue, honest at-least-once semantics, and ordering that costs only the customer who asked for it — closing on Stripe, GitHub, and Svix's own published numbers.

System design · Systems. The source ↗

A free, interactive, animated visual explainer of Design a Webhook & Trigger Delivery Platform — built to be understood, not skimmed.

Questions

How does HMAC webhook signature verification work, and why does the timestamp matter?: The sender computes an HMAC-SHA256 digest over a string built from the request, using a secret only the sender and receiver know, and attaches it as a header; the receiver recomputes the same digest and checks it matches. Stripe's own docs give the exact recipe: the signed string is "the timestamp (as a string), the character '.', the actual JSON payload," hashed with "the endpoint's signing secret as the key." The timestamp is what stops a replay: without it, a captured, legitimately-signed request could be resent forever with a still-valid signature. Stripe checks the timestamp against "a default tolerance of 5 minutes between the timestamp and the current time" and warns that "using a tolerance value of 0 disables the recency check entirely." Both Stripe and GitHub also insist the two signatures be compared with a constant-time function, never a plain ==, because an early-exit string comparison leaks how many leading bytes matched through response timing.
What retry schedule do production webhook platforms actually use, and why is jitter necessary?: They all use exponential backoff — retries spaced further apart the longer something stays broken — but the exact shape differs. Svix publishes its ladder outright: "Immediately, 5 seconds, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours, and 10 hours," eight attempts summing to about 27.6 hours from first failure to last. Stripe instead commits to a duration: it "attempts to deliver events to your destination for up to three days with an exponential backoff in live mode." Jitter — randomizing each retry's timing around its nominal gap — matters because without it, every endpoint that failed at the same moment (a shared load-balancer blip, say) retries in lockstep: all their 5-second retries land in the same instant, all their 5-minute retries land in the same instant, and so on — a synchronized "thundering herd" hitting infrastructure that just came back online and hasn't warmed up.
Why can't webhook delivery be exactly-once, and what should a receiver actually do about it?: Because the sender can never fully distinguish "the endpoint failed to process this" from "the endpoint processed it but the success response got lost in transit" — from the sender's side those look identical, so the only safe move is to retry, which means the same event can arrive twice. Stripe states the honest contract directly: "webhook endpoints might occasionally receive the same event more than once. You can guard against duplicated event receipts by logging the event IDs you've processed, and then not processing already-logged events." That is at-least-once delivery: never silently lost, occasionally duplicated. The fix lives entirely on the receiving end — give every event a stable unique ID and make processing idempotent (applying the same event twice has the same effect as applying it once), typically by checking that ID against a log of what's already been handled before acting on it again.
How do you stop one dead customer endpoint from degrading delivery to everyone else?: By making isolation structural rather than a shared resource everyone hopes not to abuse: give every registered endpoint its own queue, or a per-endpoint token bucket — a small allowance of sends that refills steadily and is spent one token per request. When one endpoint's bucket empties (because it's down or its retries are piling up), only that endpoint's deliveries wait; every other endpoint's bucket keeps refilling and keeps sending, untouched. This is what actually answers the "ordering vs. throughput" tension too: a customer who demands strict per-endpoint ordering forces serialization, but only within their own isolated queue — their slow server costs them latency, never the other thousands of endpoints sharing the platform. Left unfixed — one shared queue for every destination, delivered strictly in order — a single dead endpoint's retries create head-of-line blocking that delays healthy endpoints stuck behind it in the same queue, which is the naive design's failure mode reappearing one layer down.

Design a Webhook & Trigger Delivery Platform

Questions

Related explainers