Why can’t you just store a table as a folder of Parquet files on S3?

Because a cloud object store is a key-value store, not a database, and it can only change one object atomically. Any real change touches many files at once — a GDPR delete strips one user out of a hundred Parquet files, an update rewrites a partition — and the bucket has no bracket around those separate writes. The Delta Lake paper states the consequence directly: “because multi-object updates are not atomic, there is no isolation between queries… readers will see partial updates as the query updates each object individually.” Worse, a directory listing itself is only eventually consistent, so “a LIST after a PUT might not return the new object,” and a writer that crashes mid-change leaves the table in a corrupted state with no version to roll back to. A folder of files gives you cheap storage and append-only scans, but none of the atomicity, isolation, or rollback a table needs.

How does Delta Lake make a commit atomic on object storage?

It moves the source of truth out of the data folder and into a write-ahead log — an ordered sequence of small JSON files in a _delta_log directory, each one commit, each “an array of actions to apply to the previous version of the table.” A change can touch a hundred Parquet files, but committing it is a single act: writing one log file named for the next version number. That one file appearing makes all hundred data files live at once; until it exists, none of them are. The whole scheme rests on that create being atomic and exclusive — a put-if-absent, where only one writer may win the next version’s name. Not every store offers it: “Google Cloud Storage and Azure Blob Store support atomic put-if-absent operations,” but “Amazon S3 does not have atomic ‘put if absent’ or rename operations,” so Delta uses “a separate lightweight coordination service to ensure that only one client can add a record with each log ID” — needed only for log writes, so its load is low.

What are checkpoints in a Delta transaction log, and why do they exist?

A checkpoint is a Parquet file that stores the replayed result of the log up to a point — the whole live file set, already folded into one queryable object. They exist because a busy table accumulates enormous numbers of tiny JSON commits: a stream committing once a minute produces 525,600 in a year, and a cold reader with nothing cached would have to LIST and replay every one just to learn the current table. With a checkpoint, the reader consults a small _last_checkpoint pointer, jumps to the latest checkpoint, and only reads the handful of commits since. Delta’s clients “write checkpoints every 10 transactions,” and the paper notes “finding the set of objects to read for a query is nearly always faster using a Delta Lake checkpoint than using LIST operations.” The interval is a real dial — check in often for cheap cold reads, rarely for cheap writes.

How does optimistic concurrency handle two writers committing at once?

Both writers read the same current version, prepare their data files, and try to create the next version’s log file. The put-if-absent guarantees exactly one wins the name; the loser’s commit fails and it must decide whether it can retry. The answer depends on what it read. A blind append — an INSERT that reads nothing and only adds files — commutes with anything, so it reuses its files and retries automatically into the next version. A read-modify-write is safe to retry only if the winning commit didn’t touch the files it read; if the winner removed the very files this writer read, its change was computed against a table that no longer exists, and that conflict is real and surfaced to the user. This is why the design scales: the common case, appends, never conflicts, so the write path stays lock-free.

What’s the difference between Delta Lake, Iceberg, and Hudi?

They make the identical bet — the metadata is files sitting next to the data, and a commit is one atomic operation — and differ in the shape of that metadata. Delta Lake keeps an ordered log of JSON commits in _delta_log plus Parquet checkpoints, and commits by creating the next N.json via put-if-absent. Apache Iceberg keeps a metadata tree (table metadata → manifest list → manifest files) and commits by an atomic swap of the table’s metadata pointer — “an atomic swap of one table metadata file for another provides the basis for serializable isolation.” Apache Hudi keeps a timeline of commit files plus a record-level index and leads with fast upserts, letting you pick copy-on-write or merge-on-read. Same design, three dialects: metadata-as-files, atomic commit, and snapshots that make time travel free.

Design a Transaction Log on Object Storage

A cloud bucket can swap one file atomically and never many at once — so how does a pile of Parquet files on S3 become an ACID table? The Delta-shaped question, built up from first principles: the naive folder and the three ways it dies, the write-ahead log as the source of truth, the put-if-absent commit (and why S3 needs a tiny coordinator), optimistic concurrency with the conflict rules raced live, checkpoints against a computed cold-read cost, compaction, time travel, and the log doubling as a message queue — then an honest Delta vs Iceberg vs Hudi.

System design · Systems. The source ↗

A free, interactive, animated visual explainer of Design a Transaction Log on Object Storage — built to be understood, not skimmed.

Questions

Why can’t you just store a table as a folder of Parquet files on S3?: Because a cloud object store is a key-value store, not a database, and it can only change one object atomically. Any real change touches many files at once — a GDPR delete strips one user out of a hundred Parquet files, an update rewrites a partition — and the bucket has no bracket around those separate writes. The Delta Lake paper states the consequence directly: “because multi-object updates are not atomic, there is no isolation between queries… readers will see partial updates as the query updates each object individually.” Worse, a directory listing itself is only eventually consistent, so “a LIST after a PUT might not return the new object,” and a writer that crashes mid-change leaves the table in a corrupted state with no version to roll back to. A folder of files gives you cheap storage and append-only scans, but none of the atomicity, isolation, or rollback a table needs.
How does Delta Lake make a commit atomic on object storage?: It moves the source of truth out of the data folder and into a write-ahead log — an ordered sequence of small JSON files in a _delta_log directory, each one commit, each “an array of actions to apply to the previous version of the table.” A change can touch a hundred Parquet files, but committing it is a single act: writing one log file named for the next version number. That one file appearing makes all hundred data files live at once; until it exists, none of them are. The whole scheme rests on that create being atomic and exclusive — a put-if-absent, where only one writer may win the next version’s name. Not every store offers it: “Google Cloud Storage and Azure Blob Store support atomic put-if-absent operations,” but “Amazon S3 does not have atomic ‘put if absent’ or rename operations,” so Delta uses “a separate lightweight coordination service to ensure that only one client can add a record with each log ID” — needed only for log writes, so its load is low.
What are checkpoints in a Delta transaction log, and why do they exist?: A checkpoint is a Parquet file that stores the replayed result of the log up to a point — the whole live file set, already folded into one queryable object. They exist because a busy table accumulates enormous numbers of tiny JSON commits: a stream committing once a minute produces 525,600 in a year, and a cold reader with nothing cached would have to LIST and replay every one just to learn the current table. With a checkpoint, the reader consults a small _last_checkpoint pointer, jumps to the latest checkpoint, and only reads the handful of commits since. Delta’s clients “write checkpoints every 10 transactions,” and the paper notes “finding the set of objects to read for a query is nearly always faster using a Delta Lake checkpoint than using LIST operations.” The interval is a real dial — check in often for cheap cold reads, rarely for cheap writes.
How does optimistic concurrency handle two writers committing at once?: Both writers read the same current version, prepare their data files, and try to create the next version’s log file. The put-if-absent guarantees exactly one wins the name; the loser’s commit fails and it must decide whether it can retry. The answer depends on what it read. A blind append — an INSERT that reads nothing and only adds files — commutes with anything, so it reuses its files and retries automatically into the next version. A read-modify-write is safe to retry only if the winning commit didn’t touch the files it read; if the winner removed the very files this writer read, its change was computed against a table that no longer exists, and that conflict is real and surfaced to the user. This is why the design scales: the common case, appends, never conflicts, so the write path stays lock-free.
What’s the difference between Delta Lake, Iceberg, and Hudi?: They make the identical bet — the metadata is files sitting next to the data, and a commit is one atomic operation — and differ in the shape of that metadata. Delta Lake keeps an ordered log of JSON commits in _delta_log plus Parquet checkpoints, and commits by creating the next N.json via put-if-absent. Apache Iceberg keeps a metadata tree (table metadata → manifest list → manifest files) and commits by an atomic swap of the table’s metadata pointer — “an atomic swap of one table metadata file for another provides the basis for serializable isolation.” Apache Hudi keeps a timeline of commit files plus a record-level index and leads with fast upserts, letting you pick copy-on-write or merge-on-read. Same design, three dialects: metadata-as-files, atomic commit, and snapshots that make time travel free.

Design a Transaction Log on Object Storage

Questions

Related explainers