Show HN: Penca – branchable, versioned OLTP+OLAP on one open copy of your data

jnbh1 pts0 comments

Hi HN!This is an early proof of concept of a branchable, versioned OLTP + OLAP database that runs on a single, open copy of your data in object storage. If you are familiar with Databricks LTAP (https://www.databricks.com/company/newsroom/press-releases/d...) announcement from June, you can think of this as aspiring to be a fully open source, Apache 2.0 LTAP alternative with additional data versioning/auditability guarantees that enable audit, as_of queries, and (eventually) revert straight out of the box.How it works:1. Writes land in vanilla postgres which functions as an ephemeral hot tier2. A background pass flushes committed rows out to columnar files in an object storage cold tier3. A DataFusion based query engine merges results across the two tiers (with some fancy caching and indexing)For a deep dive on the architecture/intended design/vision see: https://penca.io/blog/the-penca-architecture.html.This project is very, very early on. It is mostly a PoC right now and there are many bugs and shortcomings with many major items still on the roadmap. I hope to get some real performance numbers soon.A few features I would like to add in no particular order:- Iceberg export- Branching (from any branch, not just main)- Improved SQL support (e.g. multi-statement execution)- Full-text search and vector indexes- Configurable isolation levels (currently last write wins)- A pgwire frontendAny and all feedback is appreciated. I d also like help! If any of this appeals to you and the project doesn t seem like the craziest idea of all time, shoot me an email at nico@penca.io.Disclaimer: before someone comments this, yes, the code is heavily AI generated. I have tried to leave as much of the dev tooling in the repo as possible to demonstrate what the end-to-end development workflow looks like. Also, the majority of the commit history is missing as I had to rename/migrate + scrub the repo prior to open sourcing.

penca open databricks https like data

Related Articles