What is a write-ahead log and how do you implement one?

A write-ahead log (WAL) is an append-only file you write a durable record to before you apply a change anywhere else. The minimal implementation is one small module: frame each record as a fixed 8-byte header (a 4-byte length and a 4-byte CRC32 of the payload) followed by the payload bytes; append() packs that frame, writes it, and — per your fsync policy — forces it to disk; replay() reads frames back in order, verifies each CRC, and stops at the first torn or corrupt one. A tiny key-value store built on top logs every set/delete to the WAL first, so its state can always be rebuilt from the file.

Why append to the write-ahead log before applying the write?

Because the durable record must exist before the in-memory change, so a crash at any instant leaves something you can replay. If you applied first and logged second, a crash between the two would lose a write that already looked applied, with no record it ever happened. Appending first inverts the risk: the only writes that can be lost are ones whose log record was never finished — and those were never acknowledged to the caller. Appending is also sequential, which is the fastest thing a disk does.

How does a write-ahead log recover after a crash?

On restart you replay the log from the beginning, re-applying each record to rebuild the in-memory state you lost. Recovery reads frame by frame and stops at the first frame that is torn (cut short) or fails its CRC — everything after that point is treated as never-written, because a crash can only damage the tail of an append-only file. A careful implementation also truncates that torn tail on open, so new appends land on a clean edge and a later replay does not stop early in the middle of the file.

What is a torn write and how does the CRC catch it?

A torn write is a record that was only partially persisted when the process or machine died — the header and some payload bytes reached the file, but the rest never did. The fixed-width header lets the reader detect two of the three cases directly: a header that cannot be read in full, or a payload shorter than its declared length, are obviously torn. The CRC32 catches the subtle third case, where enough bytes are present to look complete but were corrupted or half-written: the checksum over the payload no longer matches the stored one, so the frame is rejected and the replay stops there.

What does fsync do in a write-ahead log and when should you call it?

A plain write only lands in the operating system’s page cache; fsync forces those bytes to the physical disk, which is what actually makes a record survive a power loss. It is the durability dial. Fsyncing on every append is safest — zero acknowledged writes at risk — but caps throughput near the disk’s fsync rate. Never fsyncing is fastest but risks losing every recent write on power loss. Group commit is the middle ground: fsync once per small batch, so the at-risk window is one batch. The reference module makes this a pluggable policy so the caller chooses where on the dial to sit.

A Write-Ahead Log You Can Implement in an Hour

The low-level-design round asks you to make a key-value store survive a crash — and the whole answer is one small file. Append a length + CRC32 framed record before you touch memory, fsync on the durability dial you choose, and on restart replay the log and drop the torn tail. We author the ~120-line Python module, crash it in a child process, and walk it top to bottom.

Code walk · Systems.

A free, interactive, animated visual explainer of A Write-Ahead Log You Can Implement in an Hour — built to be understood, not skimmed.

Questions

What is a write-ahead log and how do you implement one?: A write-ahead log (WAL) is an append-only file you write a durable record to before you apply a change anywhere else. The minimal implementation is one small module: frame each record as a fixed 8-byte header (a 4-byte length and a 4-byte CRC32 of the payload) followed by the payload bytes; append() packs that frame, writes it, and — per your fsync policy — forces it to disk; replay() reads frames back in order, verifies each CRC, and stops at the first torn or corrupt one. A tiny key-value store built on top logs every set/delete to the WAL first, so its state can always be rebuilt from the file.
Why append to the write-ahead log before applying the write?: Because the durable record must exist before the in-memory change, so a crash at any instant leaves something you can replay. If you applied first and logged second, a crash between the two would lose a write that already looked applied, with no record it ever happened. Appending first inverts the risk: the only writes that can be lost are ones whose log record was never finished — and those were never acknowledged to the caller. Appending is also sequential, which is the fastest thing a disk does.
How does a write-ahead log recover after a crash?: On restart you replay the log from the beginning, re-applying each record to rebuild the in-memory state you lost. Recovery reads frame by frame and stops at the first frame that is torn (cut short) or fails its CRC — everything after that point is treated as never-written, because a crash can only damage the tail of an append-only file. A careful implementation also truncates that torn tail on open, so new appends land on a clean edge and a later replay does not stop early in the middle of the file.
What is a torn write and how does the CRC catch it?: A torn write is a record that was only partially persisted when the process or machine died — the header and some payload bytes reached the file, but the rest never did. The fixed-width header lets the reader detect two of the three cases directly: a header that cannot be read in full, or a payload shorter than its declared length, are obviously torn. The CRC32 catches the subtle third case, where enough bytes are present to look complete but were corrupted or half-written: the checksum over the payload no longer matches the stored one, so the frame is rejected and the replay stops there.
What does fsync do in a write-ahead log and when should you call it?: A plain write only lands in the operating system’s page cache; fsync forces those bytes to the physical disk, which is what actually makes a record survive a power loss. It is the durability dial. Fsyncing on every append is safest — zero acknowledged writes at risk — but caps throughput near the disk’s fsync rate. Never fsyncing is fastest but risks losing every recent write on power loss. Group commit is the middle ground: fsync once per small batch, so the at-risk window is one batch. The reference module makes this a pluggable policy so the caller chooses where on the dial to sit.

A Write-Ahead Log You Can Implement in an Hour

Questions

Related explainers