One writer, twelve workers, zero rows — Chovy's Blog

One writer, twelve workers, zero rows

2026-08-19, by Anthony “chovy” Ettinger.

How this was written: drafted with an AI assistant from my own notes,<br>then edited by me.

My feed crawler stopped. Not slowed down, stopped. It reads something like 55,000 feeds a<br>day into SQLite, and one afternoon the writes just queued up behind each other and nothing<br>ever landed. Reads were fine. A select 1 came back in 100ms all day while the<br>crawler sat there emitting one feed every few minutes.

SQLite allows one writer at a time. Everybody knows this. That is the storage engine, not<br>a setting you forgot to flip. So I assumed I had simply outgrown it and started looking for<br>the thing that was making my one writer slow.

I posted the question at<br>bufferoverride.com/q/3,<br>mostly so the answer would exist somewhere the next person can find it. Here is what a day<br>of measuring actually turned up.

Everything I blamed first, and why none of it was the cause

The database was 14GB, about 10GB of it stored article HTML. Obvious culprit. I tested<br>it: one insert took 168 seconds, one hundred inserts took 23.7 seconds. Bytes were not the<br>currency. I kept the column.

Then the full text search triggers, because writing to an FTS index on every insert<br>sounds expensive. Insert with do nothing, with a guarded do update,<br>and against a table with no FTS at all: same time. Then batch size, 1000 rows down to 25,<br>no change. Then the hosting quota, which was genuinely blown at 924% of rows read. I paid<br>to fix it and write latency moved not at all. Then the shared database group, which I<br>tested by timing an idle sibling database at the same instants: 370ms while mine took 181<br>seconds. The group was healthy.

Five theories, five days worth of plausible, all wrong.

The measurement that took one minute

Three lines, run against the live database:

db.execute('update feeds set x = x where id = ?') -> 389ms<br>db.batch([that same statement], 'write') -> 302s, FAILED<br>db.batch([20 of them], 'deferred') -> 123s, SQLITE_BUSY

The same single statement is fast on its own and times out inside a transaction. That is<br>not a slow database. That is contention, and once you see it in that shape the rest is<br>obvious.

batch(stmts, 'write') opens an explicit transaction. My poller ran four crawl<br>workers plus a card pass, a cluster pass, an author pass and an alert pass, and every one of<br>them opened its own. A dozen transactions were competing for a lock exactly one of them<br>could hold. They did not politely queue. Each waited out the client's 300 second header<br>timeout, gave up, retried, and joined the back of the pile again. Throughput was not<br>degraded, it was zero, while a plain single statement write kept answering in 389ms.

The fix is a queue with one worker

The rule I needed was simple: one write at a time for the whole system, no matter how<br>many things want to write. So I stopped letting the crawl workers touch the database at all.<br>They fetch, they parse, and they push a job.

It is a BullMQ queue running on Bun, and the entire fix is the worker concurrency:

new Worker('feed-writes', async (job) => {<br>await storeCrawl(job.data)<br>}, { connection, concurrency: 1 })

That one line is the lock. SQLite gets exactly one writer because there is exactly one<br>consumer, and it is Redis holding the line instead of a database timing out at 300 seconds.<br>Producers are as parallel as I want them to be.

What I did not expect was how much else came along with it. Backpressure became a number<br>I can look at, so queue depth tells me I am behind instead of a wall of client timeouts<br>telling me nothing. Retries with exponential backoff are already there, and a write that<br>fails lands in a failed set I can inspect and replay rather than disappearing into a catch<br>block. Fetching decoupled from writing, so I could raise fetch concurrency again without<br>adding a single writer.

Bun is doing nothing exotic here, it just runs the worker and starts fast enough that I<br>stopped thinking about the process. The whole thing is a few dozen lines.

The other half of the fix was not writing as much in the first place. I sampled 400<br>active feeds and only 11.8% had posted anything in the last week, while 15.8% had posted<br>nothing in two years, and I was re-reading all of them every hour. Scheduling each feed on<br>its own publishing rhythm took demand from 62,700 crawls an hour to about 700. A queue with<br>one worker keeps you correct. Crawling less is what made it fast.

What I would tell myself

The one writer limit was never the problem. Writing to it from twelve places at once was.<br>If your single row write is fast and your transaction hangs, stop reading query plans and go<br>count how many things in your process can open a transaction at the same time.