Starting from scale in the design of the AT Protocol - Paul's Leaflets<br>Paul's Leaflets<br>Starting from scale in the design of the AT Protocol<br>Bluesky was founded in 2022 to produce a protocol which Twitter could adopt. The product goals changed, but the requirements didn't.<br>Paul Frazee
July 28, 2026
Share
This is the blogpost of a talk I gave at DWeb Camp 26 and LocalFirstConf 26.
I’ve long been a believer in the local-first movement, an approach to software design in which user data lives on personal devices. Prior to Bluesky I worked on a number of local-first projects. These included SSB, a p2p social network, and Beaker Browser, a p2p web browser.<br>I’m a local-first believer because I’m an advocate for open source and the web. Most of my professional career has been about open computing in some form. The local-first movement has codified 7 ideals and I like them a lot.<br>Now I work on AT Protocol, a network built with the goal of “locking open” the information commons.<br>However, the protocol is a departure from local-first. Where we previously might have used peer-to-peer networks, in this case we chose to rely on federated servers.
Why didn’t we keep using local-first designs? It had to do with our needs for scale. We knew that this project needed to handle much larger systems than we’d ever built before. We had to change our approach to the problem.<br>For my background, moving away from local-first felt like heresy. What’s interesting is that by setting those new requirements, we ended up finding ways to preserve what mattered to us — openness and user autonomy — while meeting those new design goals.<br>This is how atproto became a high-scale open network.
It’s a distant memory now, but Bluesky was founded to produce a protocol which Twitter could adopt. None of us came from Twitter — we came from the dat & ipfs world — but this was the contract we were hired to complete.<br>We simplified these requirements into the phrase “no steps backwards.” We knew we had to create something which felt extremely natural to Twitter users, and which could handle the scale of Twitter without sacrificing the network’s open purpose.<br>We also wanted to make something bigger than the Twitter use-case. The Internet is a global information commons. It is inherently a high-scale endeavor. We want a world where you can write a query against that commons and get streams of useful information.
Before we could do that, we needed to look at the job in front of us. Here’s the numbers we were given:
The Twitter of 2022 wasn’t the largest service on the Internet, but it wasn’t a joke either. Six thousand tweets per second likely undersells the volume of events by at least 10x, based on the post-to-like ratio that we see on Bluesky. That’s a high volume of events.<br>The Twitter model is extremely dedicated to realtime delivery of public posts. Just serving a timeline involved a complex mesh of services.
Reasonable people can disagree about the merits of microservices, but this design isn’t significantly different than most high scale applications. Different tasks need different solutions, and breaking into multiple services can help isolate the load and specialize the solutions to specific needs.<br>This is a simplified version of the diagram:
At 240M DAU, you get scary red lines.<br>Driving all requests into a single database will saturate it. You can scale the db into a cluster or invest in sharding techniques, but if your system can accept non-transactional replication then it’s reasonable to redistribute the data into each service’s disk. This helps isolate read load between tasks.<br>Purpose-built services also help to optimize for specific needs. The userdata cluster’s job is to keep the data durable and available. The user recommendations server needs to answer graph questions. The ads server needs to handle statistical aggregations. Each of these benefits from a different set of on-disk structures over the same canonical data.<br>You handle this with data replication.
The general wisdom is to use Kafka for something like this instead of a message broker like RabbitMQ. There’s nothing wrong with message brokers, but they’re better for work scheduling than they are for data replication. Logs offer durable replays with minimal coordination between the services.
With logs, we can sync events and datastores across the datacenter and get the benefits if isolated, purpose-built systems. There’s a lot more at play when handling scale, but we’ll checkpoint here to focus on these specific properties of a high-scale system:
We’re using a variety of services which decouple read workloads from each other as much as possible.
We maintain a canonical userdata store which we then replicate to a variety of secondary databases.
We project that data into more purpose-fitting stores (like graph databases or OLAPs).
With that established, let’s look again at our protocol needs. We’ll create a triforce symbol to represent a single service:
Our goal is to get...