Designing Docs for AI Agents: What Changes and Why — The README<br>For AI agents: the site index is at /llms.txt and the full site content at /llms-full.txt. For a markdown version of a page, append .md to its URL or request it with Accept: text/markdown. llms.txt<br>Last week, Ben Swerdlow at Freestyle published Designing APIs for Agents, arguing that the API design rules we spent twenty years learning are backwards for AI agents. Agents read everything, so explicitness beats convenience, and precise errors beat forgiving defaults.<br>He's right, and the same inversion has already hit documentation.<br>Most docs teams are just coming to realize this.<br>We run a docs platform, so we know a few things about docs. Here's what's we've learned: write everything down, make every page stand alone, serve markdown instead of chrome, and treat your error messages as first-class content.
Key Takeaways
In July 2025, Cloudflare measured Anthropic crawling roughly 38,000 pages for every visitor it referred back. Docs are being read at machine scale, by readers who never see your site.
The same docs page costs 31x more bytes as HTML than as markdown. Chrome is a token tax on every agent that reads you.
Progressive disclosure, the core of human docs design, actively hurts agents.
You don't need separate docs. You need more entry points into the same content: llms.txt, per-page .md endpoints, and an MCP server.
Your Reader are Now AI Agents<br>Documentation traffic is shifting from people to machines, fast. From May 2024 to May 2025, AI and search crawler traffic grew 18%, with OpenAI's GPTBot alone up 305% (Cloudflare, 2025).<br>Need more convincing? In July 2025, Anthropic's crawlers fetched about 38,000 pages for every one human visitor they referred back; OpenAI's ratio was around 1,100 to one (Cloudflare, 2025).<br>Machines now read orders of magnitude more of your documentation than people do.
Meanwhile the humans moved to asking chat/AI agents instead of their own research. By 2025, Stack Overflow was receiving about as few questions per month as it did in 2009, the year it launched (The Pragmatic Engineer, 2025). Did the developers stop having questions? Nope, they started asking assistants, and the assistants ask your docs.<br>So the question isn't whether to design docs for agents. Agents are already your highest-volume audience.<br>The question is whether they can use what they find.<br>The Playbook We All Learned<br>Good docs for humans follow certain rules, which can be summarized as "Don't make me think". For example, start with a quickstart guide, everyone loves help videos, and hide the complex flags until the reader needs them. You also want to have everything well organized in a side nav so I can quickly find what I'm looking for.<br>Every one of those rules exists because human attention is scarce. Your users are busy, get bored in minutes, and leave the moment they feel lost (or assume this isn't what they are looking for). Traditionally, yours docs serve two purposes: technical/informational data and marketing. The former is self-evident, but marketing you might have not considered.<br>We wrote a whole post on writing documentation developers actually read, and if you compress it to one idea, it's attention management: respect the reader's time, surface the likely next step, get out of the way.<br>Design for the reader's scarcest resource, in other words. For humans, that's attention.<br>Agents Break Every Assumption<br>An agent's scarce resource is context tokens: what it can afford to fetch and hold while answering. Attention never enters into it.<br>When agents arrive at your site, they aren't navigating, click though your sidebar or typing in your search box, they are fetching. An agent lands on one URL, cold, and either finds the answer there or fetches again.<br>Agents read everything you give them, instantly. Swerdlow's phrase for APIs was "agents can read our entire docs in one sitting." Hiding complexity from them doesn't reduce confusion as it does for people.<br>And agents arrive opinionated. Their training data contains your 2025 docs, your deprecated auth flow, your old parameter names. Live documentation is the correction mechanism for those stale priors. If the current truth isn't written down and fetchable, the agent uses the old version.<br>One caution before you swing the other way: reading everything isn't free. In July 2025, Chroma tested 18 frontier models and found every single one degraded as input length grew, a failure mode they named context rot (Chroma Research, 2025), so don't make one giant page of all your docs.<br>Use Your Words<br>Give agents the words without the website by proving them with markdown (md file). We measured our own quickstart both ways, and you can rerun this today:<br>$ curl -sLo /dev/null -w "%{size_download} bytes\n" https://jamdesk.com/docs/quickstart<br>225527 bytes
$ curl -sLo /dev/null -w "%{size_download} bytes\n" https://jamdesk.com/docs/quickstart.md<br>7246 bytes
That is 31x fewer bytes, and another example is...