The Database Zoo: Exotic Data Storage Engines

The Database Zoo: Exotic Data Storage Engines

Fri Sep 19 2025

database zoo

databases

Introduction / Context

Over the past two decades, the landscape of data has changed dramatically. Traditional business records and transactional data have been joined by an explosion of new formats:

Metrics and logs from monitoring systems and IoT devices.

Embeddings and high-dimensional vectors powering modern machine learning and recommendation engines.

Social graphs capturing billions of relationships between users, products, or events.

Event streams representing continuous flows of transactions, sensor readings, and interactions.

Geospatial data from GPS devices, maps, and location-aware applications.

This diversity of data types has created new challenges for engineers. Not only are the volumes unprecedented, but the access patterns are highly varied. Some workloads demand low-latency writes at massive scale, others require complex relationship queries across interconnected records, while others rely on fast aggregations over billions of points.

For decades, the default answer to data storage was a relational database. Later, the NoSQL movement expanded the options with document stores, key-value engines, and wide-column databases. These systems addressed important needs like horizontal scalability and flexible schemas. Yet, they remain general-purpose. They are optimized for a broad spectrum of problems but often fall short in niche scenarios where specialized performance or query models are required.

This is where specialized databases come in. Built for a specific type of data and workload, these systems are not intended to replace general-purpose databases but to complement them. Each one makes deliberate design choices: storage formats, indexing strategies, compression techniques, and query execution models tuned for its domain.

This series, The Database Zoo: Exotic Data Storage Engines, explores these specialized systems in depth. Each post will unpack:

The problems a particular database type is designed to solve.

The internals: storage structures, indexing methods, and algorithms that power it.

The query models and optimizations that make it effective.

Real-world use cases and examples where these systems shine.

By the end of the series, you'll have a clearer picture of why these engines exist, how they work, and when they might be the right tool for your system.

But before diving into the specialized databases, it's essential to understand the foundations laid by SQL and NoSQL systems. We'll start with a deep dive into their history, architecture, strengths, and limitations to set the stage for why the database zoo has grown so diverse.

SQL Databases: History, Architecture, and Workloads

The relational database has been the backbone of data management for more than four decades. It's difficult to overstate its impact: most software engineers' first experience with persistent data is through SQL, and an overwhelming share of enterprise applications still run on relational systems today. To understand why specialized databases emerged, we first need to look at how the relational model was born, what problems it solved, and where its limitations lie.

The Origins of the Relational Model

In 1970, IBM researcher E. F. Codd introduced the relational model of data in his seminal paper A Relational Model of Data for Large Shared Data Banks. His idea was simple yet transformative: instead of exposing low-level storage details like files or hierarchical records, databases should represent data in a logical form based on mathematics: relations (tables), defined by rows (tuples) and columns (attributes).

This abstraction separated the what of data from the how of storage. Users could express queries in a high-level, declarative language (later standardized as SQL), while the database engine handled the underlying details: indexes, access methods, storage layouts, concurrency, and recovery.

Relational databases quickly rose to dominance because they solved a major problem of the time: data independence. Applications no longer needed to change every time the physical structure of data changed. This freed engineers to focus on business logic, while the database took care of durability, consistency, and efficient access.

Key Features of SQL Databases

Relational databases share a set of foundational characteristics:

Structured schema : Tables with predefined columns and types enforce data integrity.

SQL query language : A standardized, declarative way to retrieve and manipulate data.

Indexes : B-trees and related structures accelerate lookups and joins.

Transactions with ACID guarantees:

Atomicity : all-or-nothing operations.

Consistency : database constraints are preserved.

Isolation : concurrent transactions behave as if executed sequentially.

Durability : committed changes survive crashes.

Concurrency control : Locks, MVCC (multiversion...