Choose DuckDB rather than SQLite

rubenvanwyk2 pts0 comments

SQLite vs DuckDB on the same $16 box: every cliff moved 100x · Traceway<br>All postsEngineeringSQLite vs DuckDB on the same $16 box: every cliff moved 100x<br>Jovan Stojiljkovic·July 21, 2026<br>Same $16.49/month server, same Traceway binary, two embedded databases. DuckDB writes 4x to 15x faster than SQLite, serves dashboards at 100x the row count, and stores a billion metric points in 10.8 GB. Full numbers and methodology inside.<br>TL;DR

I spent six days working on and running a benchmark of observability data on DuckDB. I've done this with SQLite in the last blog post and I really wanted to see how DuckDB compares on a cheap CCX13 Hetzner instance. The way I've done the measurements is by implementing DuckDB as a valid storage engine for Traceway and then running its benchmarking suite. The benchmarks show that DuckDB's columnar engine is able to query 100x more data points without any major backend changes. The write throughput is also 3x to 15x higher. The result is that you can now self-host the full OTel stack that can handle a large data volume on a pretty small server. Keep reading to find out how I've done the measurements!

This is what came back:

SQLite (post 2)DuckDB (this post)ChangeMetrics writes61,712 pts/sec254,242 pts/sec4xSpans writes30,508 spans/sec95,737 spans/sec3xLogs writes4,877 rec/sec75,225 rec/sec15xMetrics read cliff1M rows (3.85 s median)100M rows (3.0 s median)100xSpans read cliff100k rows (2.27 s median)10M rows (902 ms median)100xLogs read cliff100k rows (114 ms median)10M rows (85 ms median)100x<br>A read cliff, throughout this series, is the largest table size at which real dashboard pages still load: the median of three endpoint probes at or under 5 seconds, none timing out. It earns the name because of what sits one 10x step past it: queries don't slow down, they stop coming back.

Each signal's read cliff sits exactly 100x further out on DuckDB than on SQLite, at equal or better latency, on identical hardware. I checked that symmetry against the raw JSON twice before I believed it. The box also ingested a billion metric points in under an hour into 10.8 GB of disk and stayed up, a scale the SQLite benchmark never got within 100x of. Logs, the signal post 2 said to keep off SQLite entirely, are DuckDB's biggest win.

What's actually being compared

Post 2 ended promising a ClickHouse comparison, and that post is still coming. But every time I sat down to build it, the same question got in front of me: before reaching for a client-server OLAP database, with its own container, its own memory appetite, its own failure modes, how far does an embedded one go? Traceway ships a DuckDB telemetry backend as an opt-in build, same single binary, same deployment story as the SQLite build, different storage engine under the telemetry tables. If you self-host on one cheap box, these two builds are the actual decision in front of you, and I couldn't find anyone who had published numbers for it. So I ran post 2's entire methodology against the DuckDB build and put the results side by side.

Here's the plan. Methodology first, including what changed in the harness since post 2 and one fix in the backend itself that these numbers depend on. Then writes, all three signals against their SQLite baselines. Then reads, same shape. Then the billion-row run, what queries cost past the cliffs, the fleet math for both builds, and which build I would actually run.

How I measured

The setup is post 2's, so I'll keep it short: a system under test (SUT) running the Traceway binary and a separate load generator on a private link, both in Hetzner's Nuremberg datacenter, OTLP, the OpenTelemetry wire protocol, over gzipped protobuf, uniform-random data in bounded ranges. Traceway built from commit 14b4aa6e, Ubuntu 24.04, retention off during the bench.

Two scenarios per signal. The throughput ramp : batch-size ramp at fixed rate, then a request-rate ramp at the winning batch (this time with a finer rate ladder, 1 through 25 req/sec, because the interesting cliffs turned out to sit between post 2's coarser steps), then a small-batch high-rate ramp shaped like an SDK fleet. A step passes at under 5% errors and at least 70% of target rate achieved. The read-probe : fill the table to 1M, 10M, 100M, 1B, then 5B rows, and at each level load three real dashboard endpoints per signal, with a level passing when the median is at or under 5 s and nothing hits the 6 s timeout. The 5-second bar is generous on purpose and post 2's discussion of that still applies unchanged.

Three things differ from post 2's harness, all disclosed:

The loadgen grew. Post 2 generated load from a second CCX13, which SQLite's ceilings never stressed. On my first DuckDB spans run, that loadgen box died mid-ramp while the database sat at 0% errors, the traffic generator broke before the thing it was breaking. The loadgen is now a CCX23. The SUT, the box every number describes, is unchanged.

The digestion gate. DuckDB checkpoints its write-ahead log...

duckdb post sqlite median 100x rows

Related Articles