-->

We Benchmarked It | HTML vs JSON for LLMs - LJ

hack

Skip to main content Recently I made an argument and a promise. The argument: typed trees should be authored as an HTML dialect and edited by whole-tree rewrite, because models are fluent in markup and because granular mutation tools invite granular failure. The promise: we would benchmark it, publish the design before running it, and share the findings either way, good or bad. This is the companion to HTML as a Native Data Format for LLMs.<br>The findings are in. Some of the argument survived contact with the data. Some of it didn't. And the part that survived isn't quite the part I expected.<br>The Short Version<br>The whole-tree rewrite strategy wins. The HTML format, by itself, doesn't.<br>Across 8,000 scored runs (200 procedurally generated tasks, five conditions, four models from three vendors, two prompt regimes, every prompt and seed committed before the first scored call) rewriting the whole artifact beat granular mutation tools by 5.3 points overall and by 33 points on multi-turn tasks where the agent had to reference a node from its own earlier output. But a scrupulously fair JSON twin of our setup (same grammar, same validator strictness, same error quality) rewrote trees just as accurately as the HTML dialect did. The win belongs to the strategy. The format is a wash on accuracy, and earns its keep elsewhere.<br>That's a mixed result for the article, and exactly the kind we committed to publishing.<br>Task success by tree size, pooled over four models with parity prompts; whiskers are Wilson 95% intervals.<br>How We Kept It Honest<br>A benchmark like this is easy to rig by accident. The conventional arm gets a lazy validator, terse error messages, a schema the model has to guess at, and the "novel" approach wins by forfeit. So the JSON twin was built as the single most important fairness artifact in the project: a validator exactly as strict as barkup's, emitting the same issue codes, the same message wording, the same human-readable paths, cross-checked against an independently compiled JSON Schema in the test suite. The mutation-tool arm got real tools with realistic errors. The JSON Patch arm got a battle-tested RFC 6902 implementation so it could never lose to a bug in our patch engine.<br>Five conditions, one grammar semantics: A HTML plus whole-tree rewrite (the barkup approach), B JSON plus whole-tree rewrite, C JSON plus granular mutation tools (the textbook approach), D HTML plus the same tools, E JSON Patch. Hypotheses, prompts, corpus seeds, and grading were pre-registered by commit. Graders have their own unit tests. Models: claude-sonnet-4.5, gpt-5.4, gemini-3.5-flash, claude-haiku-4.5, temperature 0. Total damage: about $225 of API spend.<br>Claim by Claim<br>"LLMs have deep, pre-trained fluency in HTML": true, but no longer discriminating. First-pass validity was at least 99.3% in every arm of every model, HTML and JSON alike. In 2026, frontier and mid-tier models emit both formats essentially perfectly at these sizes. Format fluency is real; it just stopped being a moat.<br>"Ask for an inventory of an HTML tree and it reads the labels": not supported. On exact-answer structural reading questions, HTML and JSON serializations tied: 87.1% vs 87.9%. The labels-on-the-outside intuition did not show up as measurable reading accuracy, at least up to trees of about 190 nodes.<br>"Granular tools invite granular failure": strongly supported, with a mechanism I didn't predict. Whole-tree rewrite beat mutation tools by +5.3 points overall (p zero stale-id failures. The referenced ids always survived. What actually happened, in every one of the 110 failures we audited, is that the smaller models simply never executed the follow-up edit. They made unrelated tool calls, or inserted a duplicate node instead of mutating the one they had just created, and then declared the job done. Multi-turn tool-calling is where small models quietly fall apart. A whole-artifact rewrite never exposes that surface: the edit is coherent or it's rejected, exactly as the article argued, for a reason the article didn't know about.<br>Multi-turn reference edits by model and condition. Whole-tree rewrite stays reliable; granular tools fall apart on the two smaller models.<br>And the crossover never came. I expected rewrite to win on small trees and lose to tools as trees grew and rewriting got expensive and error-prone. It didn't reverse. Rewrite led at every size we tested, up to about 190 nodes. The gap narrowed at the top of the range, so a crossover may exist somewhere beyond it, but we didn't find it.<br>"Fewer tokens burned": supported, with an asterisk. Rewrite solved small and medium tasks with 4 to 5× fewer total tokens than tools (which re-send a growing conversation on every call). At 150 nodes it was still ahead. The asterisk: rewrite tokens are output tokens, which cost about 5× more each, and providers increasingly cache the tool arms' repeated inputs, so the dollar gap is narrower than the...