How ast-grep Rewrote Tree-sitter in Rust and Made It 30% Faster | ast-grep

Skip to content

Appearance

MenuReturn to top

How ast-grep Rewrote Tree-sitter in Rust and Made It 30% Faster ​<br>Part 1 of 4 — the complete adventure<br>ast-grep rewrote Tree-sitter's C core in Rust, with AI writing the code. The new core is faster at parsing, faster at reading the completed tree, and faster in ast-grep itself. (The title's “30%” is the parser-only number; end-to-end, ast-grep runs about 22% faster.)<br>Source repository: HerringtonDarkholme/tree-sitter.<br>Two quick introductions before the numbers. ast-grep — the structural code-search tool this blog belongs to — searches code by syntax rather than by text, so every file it touches must first become a syntax tree. Tree-sitter is the parser framework that builds that tree: you give it a grammar definition, and it generates a fast parser for that language. Born in the editor world, it now powers an enormous ecosystem of grammars and tools.

Performance and peak RSS. Throughput is normalized so the unmodified C build (“C / normal”) scores 100; higher is better. RSS is peak resident memory, and the raw-parsing row shows it as a range because it varies across the benchmark's language fixtures. The outline row is ast-grep's real workload: parse every file in a repository, then walk each completed tree to extract a structural outline.<br>BenchmarkC / normalRustDifferenceRaw parsingThroughput: 100<br>RSS: 8.48–21.41 MiBThroughput: 129.74<br>RSS: 8.42–25.70 MiB+29.74% throughput<br>+20.0% RSS upper boundTree traversalThroughput: 100<br>RSS: 20.38 MiBThroughput: 110.16<br>RSS: 22.20 MiB+10.16% throughput<br>+8.9% RSSComplete ast-grep outlineUser CPU: 1.233 s<br>RSS: 26.52 MiBUser CPU: 0.960 s<br>RSS: 34.43 MiB−22.2% user CPU<br>+29.8% RSSRust won every parser and traversal fixture, and ast-grep produced exactly the same outline. Memory is the tradeoff: the Rust build uses about 8 MiB more in the ast-grep run. On the much larger TypeScript stress corpus — the TypeScript compiler repository's test-baseline tree, this project's memory torture test — it peaks at 91.2 MiB . That figure is a triumph, not a confession: earlier in the project, the same corpus peaked above 1 GiB.<br>The result is not a 1:1 replacement for upstream Tree-sitter. It is a narrower runtime built for AI coding agents that analyze complete file snapshots:<br>Existing generated languages and parsers remain compatible.<br>Native loading of WebAssembly-compiled languages and incremental old-tree reuse were removed.<br>Compatibility still requires many raw pointers and unsafe blocks.<br>That boundary keeps the grammar ecosystem useful for agentic coding while removing editor-specific machinery the target workload does not need.<br>That is the ending. Getting there was another matter.<br>Why Rewrite Tree-sitter? ​<br>Every serious ast-grep performance investigation eventually arrived at the same place: Tree-sitter .<br>ast-grep could make its rules faster. It could prune work, cache configuration, and avoid visiting irrelevant syntax. But every file still had to become a syntax tree first, and Tree-sitter built that tree. The parser was both the foundation and, increasingly, the ceiling.<br>I had dreamed about rewriting or deeply optimizing it for years. The dream usually lasted until I opened the runtime. There was a mature C implementation, binary compatibility, external scanners, error recovery, incremental parsing, ambiguous grammars, several language bindings, and a small matter of not breaking the enormous grammar ecosystem built on top of all of it.<br>For one person, this was not a weekend project. It was a Herculean task wearing a header file.<br>So nothing happened.<br>Then AI-assisted rewrite attempts started appearing everywhere—Bun, pgrust, and Roc among them. They did not prove that rewriting Tree-sitter was wise, make the runtime smaller, or make parser theory less strange. They showed that the experiment had become cheap enough for one person to attempt, giving me enough leverage to ask the unreasonable question and get an answer before the decade ended.<br>So I instructed ChatGPT to rewrite Tree-sitter's C core in Rust. The project moved from a compatibility-first translation, through a fast but unreadable optimization attempt, to a simpler runtime and real parser gains—only to discover that a faster parser could still make ast-grep slower.<br>The rest of this post follows that journey: what worked, what had to be reverted, and what it took to turn a parser benchmark win into an application win.<br>Tree-sitter's Parsing Architecture ​<br>Tree-sitter takes source code and produces a syntax tree. Each supported language begins as a grammar definition that Tree-sitter compiles into generated parsing tables and lexer code; when this series says “generated languages,” “generated grammars,” or “generated tables,” it means those artifacts. At runtime, a lexer turns characters into tokens such as identifier, +, and number. The parser then uses the generated table and a stack to decide...