Turning the database inside-out with Apache Samza — Martin Kleppmann’s blog

Skip to content

Martin Kleppmann

Student Projects

About/Contact

Supporters

Turning the database inside-out with Apache Samza

Published by Martin Kleppmann on 04 Mar 2015.

This is an edited and expanded transcript of a<br>talk<br>I gave at Strange Loop 2014.<br>This transcript was originally published on the<br>Confluent blog. The<br>video recording<br>(embedded below) has been watched over 8,000 times. For those of you who prefer reading, I thought<br>it would be worth writing down the talk.

Databases are global, shared, mutable state. That’s the way it has been since the 1960s, and no<br>amount of NoSQL has changed that. However, most self-respecting developers have got rid of mutable<br>global variables in their code long ago. So why do we tolerate databases as they are?

A more promising model, used in some systems, is to think of a database as an always-growing<br>collection of immutable facts. You can query it at some point in time — but that’s still old,<br>imperative style thinking. A more fruitful approach is to take the streams of facts as they come in,<br>and functionally process them in real-time.

This talk introduces Apache Samza, a distributed stream processing framework developed at LinkedIn.<br>At first it looks like yet another tool for computing real-time analytics, but it’s more than that.<br>Really it’s a surreptitious attempt to take the database architecture we know, and turn it inside<br>out.

At its core is a distributed, durable commit log, implemented by Apache Kafka. Layered on top are<br>simple but powerful tools for joining streams and managing large amounts of data reliably.

What do we have to gain from turning the database inside out? Simpler code, better scalability,<br>better robustness, lower latency, and more flexibility for doing interesting things with data. After<br>this talk, you’ll see the architecture of your own applications in a new light.

This talk is about database architecture and application architecture. It’s somewhat related to an<br>open source project I’ve been working on, called Apache Samza. I’m<br>Martin Kleppmann, and I was until recently at LinkedIn working on<br>Samza. At the moment I’m taking a sabbatical to write a book for O’Reilly, called<br>Designing Data-Intensive Applications.

Let’s talk about databases. What I mean is not any particular brand of database — I don’t mind<br>whether you’re using relational, or NoSQL, or whatever. I’m really talking about the general concept<br>of a database, as we use it when building applications.

Take, for example, the stereotypical web application architecture:

You have a client, which may be a web browser or a mobile app, and that client talks to some kind of<br>server-side system (which you may call a “backend” or whatever you like). The backend typically<br>implements some kind of business logic, performs access control, accepts input, produces output.<br>When the backend needs to remember something for the future, it stores that data in a database, and<br>when it needs to look something up, it queries a database. That’s all very familiar stuff.

The way we typically build these sorts of applications is that we make the backend layer<br>stateless. That has a lot of advantages: you can scale out the backend by just running more<br>processes in parallel, and you can route any request to any backend instance (they are all equally<br>well qualified to handle the request), so it’s easy to spread the load across multiple machines. Any<br>state that is required to handle a request will be looked up from the database on each request. That<br>also works nicely with HTTP, since HTTP is a stateless protocol.

However, the big problem with this approach is: the state has to go somewhere, and so we have to<br>put it in the database. We are now using the database as a kind of gigantic, global, shared, mutable<br>state. It’s like a global variable that’s shared between all your application servers. It’s exactly<br>the kind of horrendous thing that, in shared-memory concurrency, we’ve been trying to get rid of for<br>ages. Actors,<br>channels,<br>goroutines, etc. are all attempts to get away from<br>shared-memory concurrency, avoiding the problems of locking, deadlock, concurrent modifications,<br>race conditions, and so on.

We’re trying to get away from shared-memory concurrency, but with databases we’re still stuck with<br>this big, shared, mutable state. So it’s worth thinking about this: if we’re trying to get rid of<br>shared memory in our single-process application architecture, what would happen if we tried to get<br>rid of this shared mutable state on a whole-system level?

At the moment, it seems to me that the main reason why systems are still being built with mutable<br>databases is just inertia: that’s the way we’ve building applications for decades, and we don’t<br>really have good tools to do it differently. So, let’s think about what other possibilities we have<br>for building stateful systems.

In order to try to figure out what routes we could...