A Fast Path for Fixed-Length Lists in Parquet - Gunnar Morling

Gunnar Morling

Random Musings on All Things Software Engineering

A Fast Path for Fixed-Length Lists in Parquet

Posted at Jul 22, 2026

parquet

open-source

java

performance

hardwood

Table of Contents

Parquet’s Dremel Encoding

Reading Effectively-Fixed-Length Lists Faster

Performance Gains

Summary

In its current form Apache Parquet isn’t a great fit for storing fixed-length lists, such as coordinates, RGB(A) colors, or—​an increasingly common case—​vector embeddings driving search and retrieval workloads.<br>A 768-dimensional embedding is just a list of floats that always has the same length,<br>yet Parquet’s Dremel machinery encodes it as if that length could vary from row to row, spelling out and reconstructing each vector’s structure on read.<br>That costs roughly 3× more than a purely flat columnar representation of the same data (see apache/arrow#34510).

The Parquet community has recognized this problem and is actively working on addressing it.<br>Several options for adding fixed-length list support are currently under discussion,<br>for instance in the form of a new logical type FIXED_SIZE_LIST.<br>Eventually, this should bring decoding performance of fixed-length list data to the level of a fully flat schema.<br>However, until that’s the case, is there anything we can do to narrow or even close the gap by retrieving effectively fixed-length lists in a smarter way?

Turns out, yes we can.

By examining the encoded data streams for Parquet’s definition and repetition levels it is possible to detect data pages which only contain lists of the same lengths and put these onto a fast path which bypasses the regular Dremel record reconstruction.<br>We’ve implemented this optimization for the upcoming Hardwood release,<br>with striking results.<br>The exact impact depends on the chosen Hardwood reader (row vs. column) and list length.<br>The following chart shows the results from parsing ZSTD-compressed files with two list widths, each with 128M four-byte float values.

The short 3-element lists (say, 3D coordinates) see a modest speed-up of 1.1× for the row reader and a solid 2.5× for the column reader.<br>For the larger 768-element lists (for instance, vector embeddings), the speed-up is 3.7× and 2.5×, respectively,<br>getting on par with a flat column storing the same number of values.<br>The row reader’s advantage grows with the list length: it has to materialize one list object per record, an overhead the fast path can’t remove.<br>At short lists (many records), that cost dominates and the win is minor, whereas for long lists (few records) it essentially vanishes and the fast path brings the row reader down to the flat-column floor.

In the following, we’ll take a closer look at why Parquet in its current form isn’t optimal for storing fixed-length lists and how Hardwood’s fixed-length list fast path makes up for it.

Parquet’s Dremel Encoding

First, let’s explore how exactly Parquet stores nested and repeated data.<br>The storage representation is described in the seminal Dremel paper from 2010.<br>An optional list, i.e. a list which itself can be null, of optional elements has the following schema:

optional group mylist (LIST) {<br>repeated group list {<br>optional float element;

This three-level encoding is required to tell apart null and empty lists from null elements.<br>Parquet uses the notion of definition levels to encode which elements in the serialized data are null.<br>In the case of mylist, the maximum definition level is 3:

the entire list is null (definition level = 0)

the list is present but empty (definition level = 1)

the list contains a null element (definition level = 2)

the list contains an actual value (definition level = 3)

The notion of repetition levels is used to express at which level of the (potentially deeply nested) document tree elements are repeated.<br>In our example, the maximum repetition level is 1:<br>a value either continues the list of the current record (repetition level = 1),<br>or a new record’s list starts (repetition level = 0).

As an example, let’s consider a Parquet file with four records which contain the following mylist values: [1.0, null, 1.2], null, [], [4.0, 4.1].<br>Here are the value stream as well as the definition- and repetition-level streams:

The value stream solely by itself would be ambiguous;<br>when reading it back, you wouldn’t know to which record a given value belongs and where null values sit.<br>The definition-level stream encodes the null and empty values,<br>while the repetition-level stream establishes the record boundaries.<br>This mechanism is great for encoding lists which actually are of varying lengths and which may contain null elements as well as may be null themselves.

Now let’s take a look at an example of required (i.e. non-optional) fixed-length lists of required elements (max def = 1):

The definition levels are now a constant stream of 1s;<br>while this can be encoded very efficiently using run-length encoding,<br>we’d ideally...