How to Make Your Payload CMS Site AI-Ready Right Now

Skip to main content<br>Headless CMSPayloadNextJSAI

How to Make Your Payload CMS Site AI-Ready Right Now<br>Make your Next.js + Payload CMS site AI-ready: let AI crawlers in, serve raw Markdown, and wire up @graph schema to let LLMs find and cite your content.

Artsiom Kaplich<br>JavaScript Engineer

05 Jun 2026<br>Share

Unlike Googlebot, AI crawlers flood your site with heavy traffic, look for raw Markdown, and scan for custom metadata. If you’re building on Next.js and Payload, implementing these optimizations requires a minimal overhead - but leaving it misconfigured means invisible content for LLMs.<br>The shift is already measurable. Ahrefs analyzed 300,000 keywords and found that when AI Overviews appear, the top organic result loses 34.5% of its clicks. Their February 2026 follow-up put that figure at 58%. Vercel went from less than 1% to 10% of new signups arriving from ChatGPT in six months. The traffic mix is moving - quietly, but fast.

TL;DR<br>Access first . AI crawlers are blocked by Cloudflare's Bot Fight Mode by default. Unblock search bots at the CDN level, then use robots.ts to explicitly allow search indexers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) while blocking training scrapers (GPTBot, ClaudeBot, Google-Extended). Nothing else here matters until requests reach your server.

Serve Markdown via content negotiation . Add a middleware check for Accept: text/markdown and return raw Markdown from a route handler. This cuts token overhead by ~80% versus HTML and makes your content structurally preferred by LLM pipelines — high signal, low effort.

*Link your structured data into one @graph . Three Schema types — BreadcrumbList for hierarchy, FAQPage for citable facts, and Article linked to Author and Organization — are what AI search bots actually use to assess relevance and authority. Without structured authorship, models have no signal to trust your content over anyone else's.

*Skip llms.txt unless you run a documentation site . Adoption is ~2% outside developer tooling, Google ignores it entirely, and AI search crawlers don't fetch it unprompted. Spend that time on the Markdown endpoint instead.

How Do You Let AI Crawlers Reach Your Site?<br>AI bots are blocked by default on Cloudflare’s Bot Fight Mode. You have to explicitly unblock them in Cloudflare, then split search bots from training scrapers in your robots.ts.<br>Before you optimize anything, confirm the crawlers can reach your server.<br>If you use Cloudflare, Bot Fight Mode is usually on by default and blocks GPTBot and PerplexityBot before requests reach your app. Choose the option for your plan:<br>Paid plans - flip the AI-bot toggles in Security → Bots.<br>Free plan - add a WAF Custom Rule with action Skip matching the user agents you want through.<br>Same goal either way: the request has to reach your origin before anything downstream matters.<br>Cloudflare Security → Bots panel: Bot Fight Mode on, AI Scrapers and Crawlers off.<br>On the Next.js side, you handle this in app/robots.ts. A wildcard rule lets everyone in. The smarter move is to welcome AI search while blocking AI training - surface in real-time answers on ChatGPT and Claude, without silently feeding the next round of LLM training.<br>Each major vendor now ships separate user agents for those two jobs. Split them cleanly.<br>User-AgentPurposeActionOAI-SearchBotSearch indexerAllowChatGPT-UserLive in-chat fetchesAllowClaude-SearchBotSearch indexerAllowClaude-UserLive in-chat fetchesAllowPerplexityBotConversational search crawlerAllowGPTBotTraining scraperBlockClaudeBotTraining scraperBlockGoogle-ExtendedGemini training corpusBlock<br>import { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {<br>return {<br>rules: [<br>userAgent: [<br>"OAI-SearchBot",<br>"ChatGPT-User",<br>"Claude-SearchBot",<br>"Claude-User",<br>"PerplexityBot",<br>],<br>allow: "/",<br>},<br>userAgent: ["GPTBot", "ClaudeBot", "Google-Extended"],<br>disallow: "/",<br>},<br>userAgent: "*",<br>allow: "/",<br>disallow: ["/admin", "/api"],<br>},<br>],<br>sitemap: `${process.env.NEXT_PUBLIC_SITE_URL}/sitemap.xml`,<br>};

Deploy, then check your access logs three to five days later. If you don't see successful requests from GPTBot or PerplexityBot, something upstream - like a WAF, CDN, or origin firewall - is still blocking them. It's time to audit your network configuration and locate the bottleneck.

How Do You Expose a Markdown Endpoint for Every Page?<br>Use content negotiation to serve Markdown. Return raw Markdown when bots hit your URLs with an Accept: text/markdown header.<br>Sure, AI crawlers can parse HTML. But they process plain Markdown way more efficiently - it means fewer tokens, zero layout noise, and a much cleaner structure. Cloudflare reported roughly 80% fewer tokens on one of their own posts when serving Markdown vs. HTML. That's why LLM pipelines prefer it when it's availablea. And why a .md version of every public page is the highest-signal surface you can offer them.<br>The implementation has two...