I ♥ Logs | A concise, visual retelling of Jay Kreps’ classic essay on the log — the unifying abstraction behind databases, distributed systems, and real-time data.
The Log: What Every Software Engineer Should Know About Real-Time Data’s Unifying Abstraction
A concise, visual retelling of Jay Kreps’ classic essay. All diagrams are from Kreps’ own Stanford EE380 lecture (slides).
The one-sentence version: the humble append-only log is the single abstraction underneath databases, distributed systems, data integration, and stream processing — and once you see it, you can’t unsee it.
Table of Contents
Part 1 — What is a log?
Part 2 — Logs and distributed systems
Part 3 — Logs and data integration
Part 4 — Logs and stream processing
Part 5 — Building systems with the log
The whole idea in 30 seconds
Part 1 — What is a log?
Forget application logs (the human-readable ERROR: ... lines from log4j or syslog). This is the other kind of log — the one databases and distributed systems are built on.
A log is an append-only, totally-ordered sequence of records, ordered by time.
Records are appended to the end ; reads go left → right.
Each record gets a monotonically increasing number — an offset — that acts as its logical timestamp .
That ordering is decoupled from any physical clock. Order is the point.
You already use this shape all the time — even a web server’s access_log is one:
The key insight: tables and logs are dual
A log is just a record of changes . A table is the current state after applying those changes. You can always convert between them:
Log → Table: replay every change in order and you rebuild the table.
Table → Log: capture each change as it happens and you produce the log.
It’s exactly like version control: the log is the sequence of commits/patches; the table is the checked-out working copy.
This table ⇄ log duality is the thread running through everything that follows. Databases, replication, caches, indexes, and stream processors are all just different materializations of the same log.
Part 2 — Logs and distributed systems
Databases have quietly relied on logs for decades:
Crash recovery / durability — write the change to the log first (write-ahead log), then apply it. On crash, replay the log.
Replication — ship the log to replicas (Oracle, MySQL, Postgres all do this).
Why does replaying a log work? The State Machine Replication Principle :
If two identical, deterministic processes start in the same state and receive the same inputs in the same order, they produce the same output and end in the same state.
The log is “the same inputs in the same order.” That’s why consensus algorithms — Paxos, Raft, ZAB, Viewstamped Replication — are, at their core, protocols for agreeing on the contents of a log.
Two flavors of using the log:
Style<br>What’s in the log<br>How replicas work
State-machine replication (active-active)<br>the requests / operations<br>every replica runs every operation itself
Primary-backup<br>the resulting state changes<br>leader runs the operation, followers just apply the diff
Either way, the log is the source of truth, and a replica’s position is just an offset into it. A replica that falls behind or restarts simply resumes reading from where it left off.
Part 3 — Logs and data integration
Data integration = making all of an organization’s data available in all of its systems and services.
It’s unglamorous plumbing, but it’s the base of the pyramid. You can’t do reliable analytics, ML, or “AI” if the data can’t even flow reliably first.
Two trends made this hard:
Event data exploded — clicks, impressions, pageviews, metrics, logs. Orders of magnitude more than traditional database rows.
Specialized systems multiplied — OLAP, search, key-value stores, graph indexes, batch (Hadoop), caches. Each one needs the data.
Connect N sources to M systems with point-to-point pipelines and you get an O(N×M) mess — fragile, duplicated, impossible to operate:
The fix: put a log in the middle
Every source writes to a central log. Every destination subscribes and reads at its own pace. O(N×M) → O(N+M).
This is exactly what Kafka was built for at LinkedIn. The log becomes “the pipe” between all systems, and it buys you:
A logical clock for the whole org — offsets let any two systems talk about “where they are” consistently.
Producer/consumer decoupling — consumers read at their own speed; a slow or down consumer just catches up later.
Zero-touch onboarding — a new destination system subscribes without touching any producer.
Rethinking ETL. Traditional ETL tangles two separate jobs together. Split them:
Producers publish clean, canonical data to the log (this is the org’s contract).
Destinations do only their own system-specific transforms.
Real-time transforms in the middle (sessionize, enrich, join) just produce new derived logs — which look identical to source logs to anyone downstream.
Making the log scale
A central log has to eat the entire firehose. Kafka...