Agents Attacking Documents with CRDTs | 404wolf.com
Agents Attacking Documents with CRDTs
Overview
A new approach to live collaborative AI editing. A team of small agents writes<br>code against a thin library to surgically edit the document tree, streaming<br>their changes in as human-looking CRDT peers with no Markdown or patches.
personal
Contents
Our SystemViewsThe Macro Document “Toolkit”
Following the flow
Contents<br>Our SystemViewsThe Macro Document “Toolkit”
Following the flow
Agents Attacking Documents with CRDTs
Finding a safe way to let LLMs modify your document tree in a way that is<br>natural and doesn’t totally bork the document state is a hard problem. We want<br>many timed and concurrent changes in small pieces like keystrokes, surgically<br>mutating nodes, without abruptly swapping out big chunks.<br>Our System<br>Macro documents aren’t regular markdown documents. They’re a tree (an<br>ASTree) of all the<br>common nodes (headers, lists, tables), and a bunch of awesome custom ones —<br>like our @mention node that points at other documents, document preview chips,<br>etc. AI’s really great at pumping out Markdown, but for the case of Macro<br>documents, we’re a bit beyond vanilla MD.<br>For our first pass, we took a simple approach: let the LLM only swap entire top<br>level blocks with generated hunks of Markdown that we parse right replacement<br>subtrees (we use Lexical for our editor and doc<br>representation). We thought it’d be easier for LLMs to produce quality MD than<br>to reason “bold ing this word means splitting a text node into three nodes,<br>with a bold child.” a markdown **bold** word is a natural sentence for AI.<br>This was true-ish, but the “swap the entire block” approach was much too<br>coarse, and it’d often either edit way too much, or not get the job done. We<br>eventually un-shipped the implementation.
The second time around, I put focus not just on correctness, but also on editing<br>having a more natural, definitively noncreepy human vibe. So I built a black box<br>(I love black boxes) that pretends to be a human that can edit your documents on<br>your behalf. You say POST “update my tables and checklists” and Bob (AI) Sam (AI) and Lilly (AI) pop in and start editing all over the document. It’s fun,<br>the job gets done, and the way we do it the agents have full control down to the<br>deepest leaf in your document’s node tree. Let’s get building.<br>Other people who have done similar things before. The Electricsql folks have a<br>blog about using a tool-call based<br>system<br>where the LLM operates like a human editor — it walks to a spot and “starts<br>streaming” text. One idea that I had was to just give an agent full access to a<br>real Neovim instance. But the issues with these kinds of approaches are that<br>fundamentally our documents are not flat. We need a system that properly<br>understands the tree structure of our documents.<br>Views<br>My first step was playing around with a bunch of different formats for<br>presenting documents to AI. I imagined we needed a way where we would have the<br>power to know exactly what node changed so as to emit proper CRDT events. We<br>assign a stable ID to every node, and being able to see the ID in the input and<br>output of the view we give the LLM makes it easier for us to produce a diff.<br>For example, the easiest path was just give it the raw JSON state<br>"root": {<br>"type": "root",<br>"children": [ {<br>"type": "paragraph", "$": { "id": "UR-e0WKc", "searchText": "Test" },<br>"children": [ { "text": "Test", "$": { "id": "I4LEHzZP" } } ]<br>}, ],<br>"$": { "documentMetadata": { ... }, "id": "root" }<br>}And have it produce JSON-patches to get us to a new<br>state. Or we have it produce tool calls like “insert node after” on this JSON.<br>But it turns out LLMs are absolutely terrible at producing good patches or JSON<br>diffs, and the JSON itself overloads its context with complicated structure that<br>it struggles to reason about.<br>We could use a fancy markdown-with-ID stamp view<br># Test {j79y-kQTrLC}
- Test {TmIbYUa2T0B}<br>{list YDo797VdY9Q}Where we could see what node is what if the LLM edited while keeping IDs next to<br>nodes, and inserted some sort of placeholder when it wanted to create new ones.<br>We also could use raw markdown, where we “infer” what the changes are. But<br>going down this route would mean writing a complicated parser, and having to<br>deal with serializing all of our custom nodes properly.<br>What I ended up settling on is an XML tree view<br>doc><br>p id="UR-e0WKc"><br>t id="I4LEHzZP">Testt><br>p><br>doc>It’s easy, the LLM can understand the structure, and it’s natural for AI to<br>think about things in XML. The main drawback here is that XML balloons pretty<br>quickly, and context is expensive $$<br>The Macro Document “Toolkit”<br>What I realized is that we don’t need the input format for the LLM to match<br>its output! My initial approach was to just have it edit some serialized view<br>and tell it to go crazy with sed and python, but instead I got more<br>creative.<br>Claude Code recently added “dynamic workflows” recently, which is a feature<br>where a primary agent writes a script to...