Indexing Hundreds of Terabytes of JSON for Fast Lookup in Low Memory Space · Glazer Blog
↓<br>Skip to main content
Glazer Blog
Glazer Blog
Table of Contents
Table of Contents
We work in highly regulated and constrained environments. That means the operational space we work in also requires a tailored approach to data problems. Instead of relying on scalable platforms such as Databricks, connecting AWS OpenSearch to S3 storage, or using other top-tier cloud solutions, the architecture we provide often has to be designed from the ground up around the constraints of the target environment.
This means working within strict hardware budgets while still meeting requirements for sustainable data ingestion, query performance, latency, and concurrency. The territory is often unknown, and benchmarks are necessarily best estimates until the solution reaches production.
In this case, we had to design a deliberately constrained, disk-oriented indexing architecture for searching hundreds of terabytes of JSON without materializing the dataset into a conventional search engine or data warehouse. We could not afford the usual architecture given the hardware budget, so we aggressively exploited inexpensive NVMe storage, ordered KV storage, and immutable blobs.
The Problem
For this project, we were working with JSON blobs. Each blob represented a small record, averaging 1–2 KB, with records batched into JSONL files. The total data volume was expected to be around 250 TB of raw JSON, amounting to more than hundred billion JSON objects distributed across more than one hundred thousand JSONL files.
The high-level requirements were:
Preserve the original raw data during ETL as “cold” storage.
Leave room to introduce additional data sources in the future.
Make a subset of keys from the JSON objects searchable.
Support the following search modes:
Search by a specific key (typed search).
Case-insensitive search (normal search mode).
Trailing wildcard (prefix) search.
Filtering by the record time present in the record, using from and to.
Return 1,000 records from a single request with an average round-trip time of less than 5 seconds.
The real roadblock was the available hardware budget. The data had to be stored and made searchable within the same rack, and the budget constraints made essentially any memory-heavy solution infeasible.
In this environment, that meant throwing the standard stack out of the window. We instead shifted our focus toward fast disk solutions and began designing around a low-memory operating model.
Architecture
MinIO and TiKV
We decided to use MinIO for storage and TiKV for index.
MinIO was a natural choice because of its reliability, broad adoption, and existing clients such as boto3 and minio/minio-go. We knew it could scale to large volumes while maintaining good performance. More importantly, MinIO supports requesting a byte range from a specific offset within an object.
That capability was particularly attractive for our design. Combined with fast NVMe drives, it allowed us to keep the original JSONL files intact while retrieving individual records without loading entire files.
TiKV was much less familiar territory for us. TiKV is built around RocksDB, which is well suited to disk-heavy, relatively low-memory workloads. Its cluster architecture also provides a straightforward path to scaling across multiple nodes.
We could provision similar hardware with fast NVMe drives for the TiKV cluster, with the main difference being the additional memory required for RocksDB indexes and filter blocks on each node.
Our data flow, at a high level, was:
Save the original JSON batches to the MinIO cluster as JSONL files.
For each JSON record, index the searchable key values in TiKV. We refer to this as our index format.
Store pointers in TiKV values that identify the corresponding JSON record within a MinIO JSONL file.
During a search, use the TiKV keys to locate matching records and then fetch those records from MinIO using the stored pointers.
flowchart LR<br>Sources["Data Sources"]<br>Ingest["JSONL Ingestion"]
subgraph Storage["Storage & Search Layer"]<br>MinIO["MinIO Cluster<br>Raw JSONL / Cold Storage"]<br>TiKV["TiKV Cluster<br>Search Index"]<br>end
Search["Search API"]<br>Client["Client / User"]
Sources --> Ingest<br>Ingest --> MinIO<br>Ingest --> TiKV
Client --> Search<br>Search --> TiKV<br>TiKV -->|"Record path + byte offsets"| MinIO<br>MinIO -->|"JSON record"| Search<br>Search --> Client
The core idea was to turn the TiKV keyspace into a purpose-built secondary index, while using each value as a byte-range pointer into an immutable JSONL file.
This allowed the source of truth to remain in cheap, immutable blobs. The index only needed to materialize the information required to locate a matching record.
Our initial hardware plan was:
A 4 node MinIO cluster.
A 4 node TiKV cluster, with one node configured as the placement driver.
Extract-Transform-Load
Saving the JSONL batches to MinIO...