The Mystery on DuckLake: A Time-Travelling Whodunit Story - peterdohertys.website
The Mystery on DuckLake: A Time-Travelling Whodunit Story
Published on: 7/21/2026
This post is part of my series on DuckDB:
A Dab of Duck
Full-Text Search with DuckDB
A Double Shot of DuckDB: Vector Similarity Search and Quack
The Mystery on DuckLake: A Time-Travelling Whodunit Story
Overview
As part of my journey through the DuckDB ecosystem, I’ve decided to look into DuckLake. The naming here is a bit overloaded, so bear with me: There is a DuckLake DuckDB extension but DuckLake is also an open format for using a database (henceforth known as a catalog) to manage a trove of Parquet files, which is where your data actually lives. (The reality of where data lives is slightly more complicated than this but more on that below.) Compared to other lakehouses, DuckLake uses a simplified approach which allows you to store a vast amount of data while keeping it accessible for querying and remaining compute and cost-efficient (especially on the storage side).
Lakehouses are an architectural pattern which, fundamentally, decouple database compute and storage. Traditionally, your database and its logical data volume(s) lived on the same physical hardware and were tightly coupled. If you needed to scale one side of the equation, it would affect the other; if one side went down, the other would, too. Hosted, cloud-native databases (e.g. RDS) address this tight coupling to greater or lesser extents but when dealing with large data sets, they become prohibitively expensive. Lakehouses offset this by using modern object storage solutions, like S3 or Azure Blob Storage and efficient file formats (e.g. Parquet) to shunt data storage off to an inexpensive and massively scalable solution. (NOTE: DuckLake also works with local files (i.e. for local development) and all other file systems supported by DuckDB. See: choosing_storage) The result is that your data set can “trivially” scale into the TB and PB range at a fraction of the cost of traditional solutions. The catalog database and query engine (e.g. DuckDB with DuckLake extension, Apache Spark or Trino) handle the bookkeeping and retrieval of the data objects when issuing queries.
This separation also introduces a powerful amount of flexibility. For instance, you can shut your database process down when you’re not using it and spin it back up very quickly. Meanwhile, you’re not paying for provisioned storage, IOPS, etc. RDS, in particular, also does this “helpful” thing where it automatically restarts stopped instances after seven days – ask me how I know! – and if you forget to stop the instance again, the AWS bill will begin growing again. The compute and data separation also allows for clever use cases like allowing for multiple lakehouse front-ends to query the data at the same time. I could see teams wanting to use this option to benchmark queries or to use whichever lakehouse is best suited to a particular workflow.
There are now many lakehouse formats (e.g. Delta Lake, Hudi, Iceberg, etc.) and platforms built on top of them but, in practice, running them can be slow, complex, expensive (search social media for “snowflake bill” if you’ve got time to spare) and require a non-trivial amount of infrastructure just to get started. DuckLake approached the lakehouse strategy with an eye towards simplicity and created a solution which requires significantly less infrastructure and can be run on commodity hardware – to get started, anyways.
I’ve read some blog posts and watched a few conference talks about lakehouses and DuckLake prior to writing this blog post but have no first-hand experience. So, dear reader, I invite you into my canoe for a trip out onto the lake to solve a mystery.
The Plot Thickens
One of the features which falls out of the lakehouse strategy is that you get an historical record of changes and time-travelling queries for “free”. Whenever a commit happens in the catalog database, a corresponding snapshot is created. Snapshots are just rows containing metadata in the catalog, so they’re cheap and fast compared to lakehouses which use files for snapshots and require network round trips to object storage. So, if a value in an existing row is modified (e.g. updated_at or count), the existing row is marked as stale, “copied” to a new row (where/how the changes are introduced) and bookkeeping is done to create the snapshot and mark it as the “current” state. Depending on your settings, the new row may immediately be flushed to a Parquet file or remain in the catalog (i.e. inlined) until it is flushed according to your settings or manually flushed, see: data_inlining.
This ability to time travel is a feature I’ve happily used in Django projects by way of the django-simple-history library or using Postgres patterns/hacks. Regrettably, I’ve never had the opportunity to use a database, like Datomic, which offers this functionality natively.
To make this post...