Introducing Bobbin — Tangled's Blog
Hello! It is I, Lewis, with my first Tangled post.<br>At Tangled my job involves a lot of Research & Development, by<br>which I mean fucking around and finding out. Let me tell you about one<br>such finding out, which turned into this new Tangled API.
Bobbin is a read-only, API-only<br>AppView<br>for Tangled. It serves the read side of the sh.tangled.*<br>lexicons<br>over XRPC, so a<br>client can render git repositories, issues, pulls,<br>comments, stars, & follows, without having to ping N<br>PDSes<br>for their data. It has no permanent storage. Everything it knows is only<br>ever in RAM and gets backfilled from upstream every restart.
I, along with many others, find the term "AppView" to be too<br>abstract or perhaps even unfortunately-named. In atproto, datasets<br>are distributed across Personal Data Servers (PDSes) that users each<br>own, so an AppView is a single API through which one can query<br>a dataset, in contrast to all downstream clients each maintaining a<br>connection with every PDS. An AppView is one of the many programs in<br>use to reduce the strain on PDSes, so that a PDS is relatively cheap and<br>resource-efficient to run.
Let me list some problems that I find Tangled's AppView to have, such<br>that the journey of discovery that led me to Bobbin makes sense:
Tangled's AppView has no API; it doubles as a web client via SSR.
It is brittle in the sense that it's not able to backfill<br>atproto data easily.
Its architecture from the ground up is inherently built for the kind<br>of web program that assumes it is the source of truth of data.
Because of all of the above, and the fact that there is some<br>private data such as email -> user DID<br>ties, there ought to only be one AppView for Tangled. This means that<br>latency to Tangled is vaguely a function of distance from the server<br>in Stockholm.
It is hungry for RAM during a RAM crisis.
Flipping to the inverse of all of these items would yield me my ideal<br>AppView. An API-only, easily-backfillable, easily-distributable,<br>easy-on-RAM program.
Why "API-only"? Once it's easy to retrieve Tangled data via generic<br>XRPC API, more programs can be built on top of it; programs such<br>as alternative frontends, third-party clients, one's own targeted<br>dashboards, in my case it was to enable a good CLI (though I suppose<br>that just comes under alternative frontends). The existing Tangled<br>AppView being its own web client means there's nothing extendable to<br>build on, unless one is a scraping wizard; one has to figure out their<br>own way to go grab the Tangled dataset from the world of PDSes in the<br>wild.
While I was mulling how to go about creating such a program for fun, I<br>have also been having a crisis about where to store state, after I've<br>woken up in cold sweats wondering if Tranquil PDS' homegrown embedded<br>DB could fail in new mystical ways. How could I avoid this next program<br>having to write any durable, fallible state? Besides, AppViews are<br>downstream of PDSes, which means the more we try and hold onto state in<br>an AppView, the more chance there is that it'll diverge from reality.<br>All it takes to have an AppView be wildly incorrect is missing a single<br>record, so it is of upmost importance to have an AppView be able to<br>reconcile what it thinks it knows with what's actually true.
Being the simpleton that I am, I think that there is no better way<br>to make a program whose only job is to cache and serve records than<br>to make it never touch disk in the first place. I would want it to<br>simply backfill all of its data from scratch on every restart. In that<br>case, reconciliation with reality is just a restart away and happens<br>automatically! Was that feasible? What if such a backfill took a whole<br>day every time? That would sure make deployments slow, would it anyhow<br>be worth it?
My first tests of this concept gave promising results, I<br>threw together a little server that could in fact backfill<br>all of Tangled's atproto dataset in around 15 minutes, using<br>Hydrant as its<br>aggregator. I decided that an acceptable backfill time would be under<br>5 minutes or less, so that it would be feasible to switch out running<br>instances on a blue/green deploy within a reasonable warming timeframe.
Hydrant is an aggregator for arbitrary atproto datasets that works<br>by tailing the live firehose,<br>backfilling every repo<br>on the network by pulling their<br>CARs,<br>and serving it back over a websocket. All Bobbin has to do is open a<br>connection, ask for cursor 0, and (re)build its index off the resulting<br>barrage.
Massive gratitude for<br>Dawn for having<br>made Hydrant! Then for hearing my struggles and going and optimizing it<br>also for fun!
Once I reached backfill in 5 minutes, I moved the goalpost for fun, that<br>the acceptable backfill time should now be 90 seconds or less. This is<br>around the number we are approaching today, depending on how warm the<br>upstream Hydrant instance is; ranging from 30 seconds in best-case<br>scenario to 20 minutes in a disaster where Hydrant is also pulling<br>records from PDSes from scratch.
Yes, you too can...