Introducing Dux — Christopher Grainger

Home<br>30 March 2026

elixir

dataframes

duckdb

dux

Christopher Grainger

Introducing Dux

DuckDB-native dataframes for Elixir, with distributed execution on the BEAM.

Four years ago I introduced Explorer, a dataframe library for Elixir built on Polars. I wanted the elegance of dplyr, the speed of a proper columnar engine, and the joy of working in Elixir. Explorer delivered on that. It's been adopted widely in the community, it's integrated with Livebook, Nx, and Ecto, and I'm still proud of what we built.

But I've been working on something new. Dux (GitHub, website) as in ducks, as in multiple ducks. Plus an 'x' because, you know, it's Elixir. It borrows Explorer's verb design from dplyr, but the architecture is fundamentally different. It's faster on single-node operations, it distributes natively across BEAM nodes, it has features Explorer never had (graph algorithms, ASOF joins, cross-source queries), and the whole thing compiles to SQL with no NIF layer. Let me explain why.

Why not just keep building Explorer?

Explorer is great. I mean that sincerely. Philip, José, Billy, and the community have done incredible work, and it's a mature, well-tested library. If you're doing single-node data analysis in Elixir, Explorer is excellent.

But we ran into some things that were hard to fix within Explorer's architecture.

The first is that Polars maintenance became a true albatross. Explorer wraps Polars via Rust NIFs, which means every Polars release can break the bindings. Polars is an amazing project, but the development process is fast and a lot of it is focused on the Python library. We found ourselves in a constant catch-up cycle, eventually having to give up features because we couldn't keep pace with upstream changes. And the NIF layer itself is a source of friction: Rust compilation, precompiled binary distribution, debugging across the FFI boundary. It's a lot of machinery.

Dux sidesteps all of this. It talks to DuckDB via ADBC (Arrow Database Connectivity), a pure Elixir driver with precompiled binaries that just downloads and works. Operations accumulate as an AST and compile to SQL, so the entire interaction with DuckDB is through a standard database protocol. No NIF, no Rust toolchain, no FFI. The development complexity difference is significant. And because DuckDB has a proper extension system, you can build extensions that add capabilities to every Dux user without touching the Dux codebase itself.

We also always knew the right direction was lazy-by-default: accumulate operations and only execute when you need results. But this was very difficult with Polars's Series API and the eager/lazy split. You end up fighting the abstraction.

The second is distribution. We tried to build Explorer.Remote to run Explorer on remote BEAM nodes. It petered out. The fundamental issue is that Polars operations are Rust function calls, and you can't serialise a Rust function pointer and ship it to another node. You'd need to rebuild the entire operation on the receiving side, which means duplicating the expression compiler for the remote path. It's a lot of work for something that should be natural on the BEAM.

The third is that DuckDB happened. A few weeks ago, I built a DuckDB backend for Explorer. And in doing so I saw that DuckDB would let us realise the lazy-by-default and distributed vision that we'd always wanted but couldn't make work with Polars. So I went for it.

DuckDB has quietly become extraordinary. It reads Parquet, CSV, NDJSON, and Excel natively. It pushes filters into S3 reads. It queries Postgres and SQLite via ATTACH with predicate pushdown. It has window functions, recursive CTEs, 500+ built-in functions, Iceberg and Delta support, and it runs everywhere. When DuckDB is your engine, a lot of features you'd otherwise have to build are just... there.

What Dux is

Dux is DuckDB + Elixir, compiled to standard SQL under the hood. You write:

require Dux

Dux.from_parquet("s3://data/events/*.parquet")<br>|> Dux.filter(amount > 100 and status == "active")<br>|> Dux.group_by(category)<br>|> Dux.summarise(total: sum(amount), n: count(id))<br>|> Dux.sort_by(desc: total)<br>|> Dux.compute()<br>If you've used Explorer, this looks familiar. The verb design is the same: filter, mutate, select, group_by, summarise, sort_by, join. Pipe through functions, each returns a new dataframe.

The differences from Explorer:

No Series API . All operations are dataframe-level. Column expressions use macros where bare identifiers are column names and ^ interpolates Elixir values (SQL-injection safe parameter binding).

Lazy by default . Operations accumulate as an AST until you call compute/1, to_rows/1, or to_columns/1. Data doesn't touch Elixir until you ask for it.

DuckDB is the only backend . No pluggable backends, no abstraction tax. Every DuckDB function, extension, and optimisation is available.

_with variants for raw SQL. Every macro verb has a _with suffix that accepts...