120,000 Lines of Rust: Inside the Nosdesk Backend - Kyle.au

KYLE.AU<br>PHILLIPS

Blog<br>Guides<br>Work<br>Projects<br>Notes

120,000 Lines of Rust: Inside the Nosdesk Backend<br>Published: 28 May 2026

I wrote before about why I built Nosdesk, and somewhere in that story I said I chose the hard path with Rust and it paid off. That post was about the product. This one is about what powers it.<br>What started as a handful of files has grown, over a year or so, into something close to 120,000 lines of Rust across roughly 260 modules, with around 1,030 tests holding it in place. It still ships as a single binary that comes up with one docker compose up. The stack stayed deliberately small: Actix-web on top, Diesel over Postgres for storage, Redis for fan-out, Tokio running everything underneath.<br>Three habits ended up shaping the work, and they run through everything below:<br>Push the dangerous mistakes into the type system , so the wrong thing won’t compile instead of merely being discouraged.<br>Split the pure logic from the I/O around it , so the tricky parts become functions you can test without a database or a socket.<br>Make comments explain why, not what : the alternative I rejected, the RFC I’m honouring, the bug that taught me the lesson.<br>Everything Is a Pipeline<br>When a client connects to Nosdesk, the first thing it does is pull a full snapshot of everything it’s allowed to see (bootstrap sync), which on a real workspace is a lot of rows. Load that into a Vec and serialise it in one shot and you get a workspace-sized memory spike on every connection.<br>So bootstrap is a stream. Rows are serialised as newline-delimited JSON and pushed through an mpsc::channel(64), so a slow reader back-pressures the producer instead of pinning the whole result set in RAM. Diesel is synchronous, which means the query side runs on spawn_blocking and the bytes come back through a ReceiverStream. The whole snapshot lives inside one transaction, so the client sees a consistent point-in-time view even while other writes are landing.<br>That shape repeats throughout the codebase: a bounded buffer, the blocking work pushed off the runtime, back-pressure as a feature. Once you see data flow as a pipeline where the producer can outrun its consumer, you stop writing code that falls over under load.<br>Teaching Postgres to Push<br>The sync engine is one append-only log doing three jobs. Every meaningful change in the system writes a single row into sync_actions, and three independent consumers read from that one write: HTTP delta sync for clients catching up, a live push channel for clients already connected, and the audit trail. Collapsing it into one log means a client and an audit row can never disagree about what happened. If the write landed, every consumer sees the same canonical event in the same order. The cost is one extra row per business event, which on Postgres is essentially free.<br>The harder of the three is the live push: how the server learns, in real time, that a row has just landed. Postgres has LISTEN/NOTIFY for this, but Diesel’s synchronous libpq client can’t surface async notifications cleanly, so I open a dedicated tokio-postgres connection outside the main pool, purely to listen. Its notification API is poll-based, which I wrap into a Stream:<br>let mut messages = stream::poll_fn(move |cx| conn.poll_message(cx)); That one line bridges a callback-shaped C-style API to async/await. Adapting awkward upstream APIs into the shape your system wants is most of what async Rust is.<br>The decision that load-bears the whole subsystem is that the NOTIFY is intentionally empty. It carries no payload, no row id, no hint at what changed. Every wakeup means “drain anything new past my watermark”, and the listener runs WHERE sync_id > last_seen to find it.<br>That choice looks wasteful for about thirty seconds, and then the failure modes it deletes start adding up. Fifty rows committed in one transaction collapse to one wakeup instead of fifty. A burst of writes debounces on its own. Most importantly, it stays correct under concurrent writers: a handler that trusted the payload and fetched “the row named in the notification” would silently miss the rows everyone else committed in the same window. The listener catches up on connect, drains in a loop when it hits a page cap, and reconnects with exponential backoff. The watermark lives in memory on purpose. SSE isn’t the only delivery path, so any gap a restart leaves behind gets covered by the client’s normal delta catch-up.<br>The Live Layer<br>The broadcast bus that fans the log out to connected browsers runs over Server-Sent Events. Each topic pairs a tokio::sync::broadcast sender for the live tail with a small ring buffer of recent events for replay, so a client that briefly drops its connection reconnects with the standard Last-Event-ID header and backfills the gap instead of resyncing from scratch.<br>The per-client subscription is a hand-written Stream implementation that does four things at once: it merges every topic the...