Rebundling the Database — into a Language — Cambra<br>← All posts<br>For over a decade, the fashionable direction in software has been apart. We<br>split applications into microservices. We unbundled the database into logs,<br>stream processors, and derived views. We separated storage from compute. Each<br>move solved a real problem, but each also created problems of its own. People<br>have noticed those problems: advocates for monoliths and consolidated stacks get<br>louder every year. Unfortunately, apart and together represent two ends of a<br>tradeoff, so neither of these positions is objectively better.
To break out of this unfortunate tradeoff, we need to take the database out of<br>its box, and rearchitect it as a programming language. Today,<br>Cambra — a programming language that<br>rebundles the database — is open source, under Apache 2.0, and being developed<br>in the open.
Apart, for better and worse
Some tradeoffs are fundamental, implied by the laws of physics. However,<br>software is currently caught in a false tradeoff, which arises from the<br>artificial coupling between physical resources and logical concerns.<br>Physical things need to be separated: people need to be divided into teams,<br>systems need to be modularized into components, and the demands on compute<br>resources scale independently. Unfortunately, logical concerns often span teams,<br>components, and scaling laws. We have seen this tension play out repeatedly over<br>the last decade: microservices vs monoliths, databases vs unbundled data infra,<br>colocated vs separated storage and compute.
Microservices are a response to a real problem: a hundred engineers cannot ship<br>one binary without trampling each other. Giving each team a service, with an<br>interface it owns, lets teams move independently. Unfortunately, that’s when<br>Conway’s law kicks in: service<br>boundaries align with the org chart, not the logic. Logic gets artificially<br>split into services to match an architecture diagram that struggles to evolve<br>with business needs. Every cross-cutting change becomes a multi-team<br>negotiation, and the system as a whole grows harder to evolve even as each piece<br>gets easier to ship.
Unbundling the monolithic database addressed another problem. Database machinery<br>— indexes, materialized views, replication, a query planner — is some of the<br>most broadly useful software ever written, and all of it sat locked behind a<br>wire protocol and a query language that was never designed for what it does<br>today. Jay Kreps’s The Log and Martin Kleppmann’s<br>Turning the database inside-out<br>pointed out something beautifully obvious: modularizing the components of a<br>database is extremely powerful. Unbundling let a generation of systems put the<br>database’s machinery to work outside the database’s box. But it also<br>fragmented one box into a fleet of distributed systems<br>wired together by hand. Every wire is an interface that no compiler checks, no<br>type system spans, and no debugger crosses. Even worse, the coordination the box<br>used to do quietly (atomicity, isolation, fault tolerance) became application<br>code.
Coupling storage capacity to processing capacity is also problematic: state and<br>processing have different lifecycles and different economics. Elastic compute<br>over durable, cheap, shared storage is operationally wonderful. We say this from<br>experience — we spent the last decade building these exact systems at Google and<br>at Snowflake (<br>who popularized “separation of storage and compute”).<br>But storage/compute separation tries to ignore a fundamental fact of physics:<br>smaller things are faster than<br>bigger things. Smearing your durable state across a fleet of storage nodes will<br>never be as fast as a well-designed, colocated system. Separation buys<br>elasticity and economics, never speed. You can recover some of this performance<br>with caching, but that hauls in one of the<br>two hard things. In the<br>meantime, you have to rearchitect your whole system to take advantage of the<br>improved economics.
Each of the above trends found a real problem — in the shape of organizations,<br>in the database’s implementation, in the economics of computation — and solved<br>it by modularizing software in a novel way. Unfortunately, this resulted in<br>software being pulled apart at logical seams. The benefits were organizational<br>and operational. The cost was an explosion of complexity in the application<br>layer.
Rebundling at the right layer
How do we pull these seams back together? How do we get organizational<br>decoupling without slowing development to a crawl? How can we take advantage of<br>modular database components without having to re-implement the database? How can<br>we decouple resources without cementing a specific architecture into our<br>application?
The answer won’t surprise you:
Decouple the logical structure of an application from its physical<br>representation, so that you can make the tradeoffs demanded by physics<br>without rewriting your whole application.
Model this logical structure in a way that composes cleanly, so that you<br>can modularize your...