The Long Road to Bottomless Postgres<br>Blog Home<br>The Long Road to Bottomless Postgres

Shaun Thomas|June 24, 2026

Every database eventually runs into the same wall: storage costs money, and the data nobody queries anymore costs exactly as much as the data everyone does. A five-year-old row occupies the same expensive block storage as the order that came in thirty seconds ago. Postgres doesn't know the difference, and why would it? That's honestly a common refrain among most database engines.<br>As a result, many have dreamed of fixing this by decoupling compute from storage. Push the cold, ancient data down to cheap object storage like S3, keep the hot data local and fast, and let a single query span both. Better yet, store that cold data in an open columnar format for optimal analytics. It's a vision that launched an entire cottage industry of extensions, forks, and re-architectures, each one tackling the problem in its own particular way.<br>But there are so many approaches now, and none have "the" definitive solution. Each contender seems to demand a different sacrifice, whether that's a forked binary, an extra daemon, a duplicate copy of table data, or eschewing writes entirely. Let's explore the external storage ecosystem and see what we find.<br>Perchance to Dream<br>So what does "bottomless" actually mean, and how do we get there?<br>Postgres declarative partitioning offers the first potential tool in our arsenal. The ability to split data into definitive chunks means the planner can leverage pruning to focus on the most relevant data based on our predicates. From there, we can consider purposefully tiering data based on storage cost and demands. It's not a huge stretch of the imagination to use tablespaces to further federate "hot" and "cold" storage this way. No longer would our critical transaction-sensitive data cost the same as accumulated metrics useful only to analytics.<br>It's possible to take that idea even further. In the most extreme cases, innovative users utilize Postgres Foreign Data Wrappers to present entirely remote content as local data. Some wrappers are better than others, but this method of decoupling further taps into the potential of "limitless" storage. Indeed, Postgres doesn't even really care where that data lives at all, only that the FDW reacts as instructed. Postgres Table Access Methods can even obfuscate the foreign nature of the content entirely.<br>And perhaps finally, there's the storage format itself. The Postgres data heap is great, but it's understandably tied to the engine, with all that implies. Storage isn't columnar for better analytic access, it's encumbered with reasonable atomic locking that actively resists multiple write sources, and has very real limitations. Apache Parquet or Iceberg on the other hand, have quietly become the lingua franca for data lakes of all description. Stored externally in those formats, suddenly multiple engines can interact with the "cold" data along with Postgres, in a format better suited for analytics.<br>In that dream, Postgres is a familiar, operational query shell over literally any amount of data in any location. If that's not "bottomless", I don't know what is. And yet, every project that has chased that dream has missed the mark in one way or another. How? Well, let's take a look at some of the existing projects and see for ourselves.<br>Neon to The Extreme<br>The most thorough way to decouple storage from compute is to supplant the storage engine with an entirely new one. Neon decided that was the most direct way to attain "the dream"; just put all data in S3 where "bottomless" is in the product description.<br>Neon replaces the Postgres storage manager so that pages no longer come from local disk at all. Compute nodes become stateless Postgres processes that fetch pages on demand from a separate service called the pageserver , while a redundant set of safekeepers  durably store the write-ahead log before it's processed and uploaded to cloud storage. In a way, the WAL effectively becomes the database. There's no local heap to fsync, and the log sequence number doubles as a temporal key, so the pageserver can reconstruct any page at any point in time by replaying WAL forward from a base image.<br>It's an elegant design that's entirely transparent to SQL clients. Applications have no idea that pages are being served over the network from layered files in S3. But that flexibility comes with a couple catches.<br>The first is that this requires a patched Postgres. The storage substitution is woven through the codebase. Neon must start a node without reading a checkpoint record, track the last-written page LSN, and include copious prefetching to address S3 access latency. It's not possible to drop Neon onto a stock community build; it's barely even possible to run it outside of a cloud context. It's more of an ecosystem than the Postgres most users would recognize.<br>The second catch is latency. Neon's own documentation is refreshingly candid that it "doesn't...