What does exactly-once mean in stream processing?

It means that under any failure, each input event affects the result exactly once — never lost, never double-counted. The honest version is exactly-once state plus exactly-once side-effects. A consistent snapshot (checkpoint) gives exactly-once state on recovery; getting the external sink to match requires a transactional or idempotent write. Flink’s docs put the state guarantee precisely: a dataflow “can be resumed from a checkpoint while maintaining consistency (exactly-once processing semantics) by restoring the state of the operators and replaying the records from the point of the checkpoint.”

How do checkpoint barriers create a consistent snapshot?

The coordinator injects special records called barriers into the source streams; they “are injected into the data stream and flow with the records as part of the data stream. Barriers never overtake records, they flow strictly in line.” When a barrier passes an operator, that operator snapshots its state. The union of all those per-operator snapshots is a globally consistent cut of the whole dataflow — a variant of the Chandy-Lamport distributed-snapshot algorithm — taken with no stop-the-world pause.

What is a watermark and how does it handle late, out-of-order data?

Events carry an event time (when they happened) that can arrive out of order. A watermark is a marker flowing with the stream: “A Watermark(t) declares that event time has reached time t in that stream, meaning that there should be no more elements from the stream with a timestamp t’ <= t.” It lets a window decide when it is safe to fire. A watermark set too tight drops late events; allowed lateness keeps window state around a while longer so late events still update the result.

How does a stream processor get end-to-end exactly-once with an external sink?

By tying a two-phase commit to the checkpoint. When a checkpoint starts, the sink pre-commits its output in a transaction (“The starting of a checkpoint represents the ‘pre-commit’ phase of our two-phase commit protocol”). Only after the checkpoint completes does the coordinator tell every operator to commit. If a crash lands between pre-commit and commit, recovery re-commits the pending transaction — Flink’s TwoPhaseCommitSinkFunction “always issues a preemptive commit when restoring state from a checkpoint.” A plain, non-transactional INSERT sink gets duplicates instead.

What is the difference between exactly-once state and exactly-once side-effects?

Exactly-once state is internal: after recovery the operators’ counts, windows, and aggregations reflect each event once, because they are restored from a consistent snapshot and the source replays the rest. Exactly-once side-effects are external: rows written to a database, messages emitted to Kafka. A checkpoint alone does not undo external writes that already happened before a crash, so re-processing replays them — duplicates — unless the sink is transactional (2PC) or idempotent. The whole “exactly-once” story is engineering the second to match the first.

Design a Stream Processor with Exactly-Once

Q: How does a stream processor get end-to-end exactly-once with an external sink?

By tying a two-phase commit to the checkpoint. When a checkpoint starts, the sink pre-commits its output in a transaction (“The starting of a checkpoint represents the ‘pre-commit’ phase of our two-phase commit protocol”). Only after the checkpoint completes does the coordinator tell every operator to commit. If a crash lands between pre-commit and commit, recovery re-commits the pending transaction — Flink’s TwoPhaseCommitSinkFunction “always issues a preemptive commit when restoring state from a checkpoint.” A plain, non-transactional INSERT sink gets duplicates instead.

Q: What is the difference between exactly-once state and exactly-once side-effects?

Exactly-once state is internal: after recovery the operators’ counts, windows, and aggregations reflect each event once, because they are restored from a consistent snapshot and the source replays the rest. Exactly-once side-effects are external: rows written to a database, messages emitted to Kafka. A checkpoint alone does not undo external writes that already happened before a crash, so re-processing replays them — duplicates — unless the sink is transactional (2PC) or idempotent. The whole “exactly-once” story is engineering the second to match the first.

From a single consumer with a dictionary to a dataflow of operators that survives a crash without double-counting: checkpoint barriers, watermarks for late data, a keyed state backend, and a two-phase-commit sink — drawn, computed, and animated.

System design · Systems. The source ↗

A free, interactive, animated visual explainer of Design a Stream Processor with Exactly-Once — built to be understood, not skimmed.

Questions

What does exactly-once mean in stream processing?: It means that under any failure, each input event affects the result exactly once — never lost, never double-counted. The honest version is exactly-once state plus exactly-once side-effects. A consistent snapshot (checkpoint) gives exactly-once state on recovery; getting the external sink to match requires a transactional or idempotent write. Flink’s docs put the state guarantee precisely: a dataflow “can be resumed from a checkpoint while maintaining consistency (exactly-once processing semantics) by restoring the state of the operators and replaying the records from the point of the checkpoint.”
How do checkpoint barriers create a consistent snapshot?: The coordinator injects special records called barriers into the source streams; they “are injected into the data stream and flow with the records as part of the data stream. Barriers never overtake records, they flow strictly in line.” When a barrier passes an operator, that operator snapshots its state. The union of all those per-operator snapshots is a globally consistent cut of the whole dataflow — a variant of the Chandy-Lamport distributed-snapshot algorithm — taken with no stop-the-world pause.
What is a watermark and how does it handle late, out-of-order data?: Events carry an event time (when they happened) that can arrive out of order. A watermark is a marker flowing with the stream: “A Watermark(t) declares that event time has reached time t in that stream, meaning that there should be no more elements from the stream with a timestamp t’ <= t.” It lets a window decide when it is safe to fire. A watermark set too tight drops late events; allowed lateness keeps window state around a while longer so late events still update the result.
How does a stream processor get end-to-end exactly-once with an external sink?: By tying a two-phase commit to the checkpoint. When a checkpoint starts, the sink pre-commits its output in a transaction (“The starting of a checkpoint represents the ‘pre-commit’ phase of our two-phase commit protocol”). Only after the checkpoint completes does the coordinator tell every operator to commit. If a crash lands between pre-commit and commit, recovery re-commits the pending transaction — Flink’s TwoPhaseCommitSinkFunction “always issues a preemptive commit when restoring state from a checkpoint.” A plain, non-transactional INSERT sink gets duplicates instead.
What is the difference between exactly-once state and exactly-once side-effects?: Exactly-once state is internal: after recovery the operators’ counts, windows, and aggregations reflect each event once, because they are restored from a consistent snapshot and the source replays the rest. Exactly-once side-effects are external: rows written to a database, messages emitted to Kafka. A checkpoint alone does not undo external writes that already happened before a crash, so re-processing replays them — duplicates — unless the sink is transactional (2PC) or idempotent. The whole “exactly-once” story is engineering the second to match the first.

Design a Stream Processor with Exactly-Once

Questions

Related explainers