Your site welcomes AI crawlers. Mine was serving them six words. — Smith App Studio

Share via…<br>Copy link<br>Share on X<br>Share on LinkedIn<br>Share via Email

Journal

Explore the apps &rarr;

From the Journal &middot; July 27, 2026

Your site welcomes AI crawlers. Mine was serving them six words.

Google&rsquo;s crawler runs JavaScript. ChatGPT&rsquo;s does not. I measured what that difference costs across eleven of my own production sites — and three of them were invisible.

By Dr. Maasi J. Smith &middot; 9 min read &middot; Published July 27, 2026

Key takeaways

AI crawlers do not execute JavaScript. GPTBot, ClaudeBot and PerplexityBot read your raw HTML once and move on. Googlebot renders JavaScript. They do not.

A client-rendered single-page app is therefore invisible to them — not badly ranked, invisible.

I measured eleven of my own production sites. Three served AI crawlers fewer than ten words. All three had perfect robots.txt files explicitly inviting those crawlers in.

I fixed all three with build-time prerendering: 5 &rarr; 1,442, 6 &rarr; 3,048, 8 &rarr; 3,021 words. The renderer took an afternoon. The guards that stop it shipping something worse took longer — and every one of them caught a real bug.

You can check any site in about four seconds. The script is below, and I would rather you ran it on your own site than took my word for anything here.

The two-minute version

I thought I had a content problem

Last week I asked ChatGPT to recommend an app for separated parents who need court-ready records of their communication. I have built exactly that product. It has been live for months. It did not come up.

I assumed I had a content problem. Not enough blog posts, not enough backlinks, the usual.

I had a plumbing problem. My server was sending AI crawlers an empty room.

The mechanics

What is actually happening

There are two kinds of crawler on the internet now, and they behave completely differently.

Googlebot runs a headless Chrome. It fetches your page, executes the JavaScript, waits for the framework to build the DOM, and indexes the result. This is why single-page apps have been survivable in Google for a decade.

AI crawlers — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot — do not. They issue one HTTP request, parse whatever HTML comes back, and leave. No rendering. No second attempt. No waiting.

Google&rsquo;s crawler runs JavaScript. ChatGPT&rsquo;s does not. That single difference decides whether an AI assistant can recommend your product.

This is not speculation. An analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution. Crawler-log studies show GPTBot downloads JavaScript files in roughly 11.5% of requests and ClaudeBot in about 23.8% — and neither has ever been observed running them. They pull the file. They do not open it.

So if your marketing page is a React or Vue app that boots into , here is the complete, exhaustive list of what an AI assistant knows about your company: your tag and your meta description.

That is it. That is the whole corpus.

The audit

What I found on my own sites

I run eight consumer products, a B2B platform, a Web3 marketplace and a studio site. I wrote a four-line script, pointed it at all eleven, and went to make coffee. I came back to this:

Measured 27 July 2026 using the GPTBot user-agent. Script below — please verify me.

SiteWords servedVerdict

FamilyCare.Help5Invisible<br>CoParent.Help6Invisible<br>PostPilot.Help8Invisible<br>Inmigrante.Help2,479Fine<br>FamilyCare Facility2,523Fine<br>Nexaria Digital2,276Fine<br>ExamPilot.Help2,212Fine<br>Zari.Help2,178Fine<br>PayLess.Help2,116Fine<br>Smith App Studio2,120Fine<br>ProfilePhoto.Help1,561Fine

Three products. Five, six, and eight words.

Now here is the part that stung. I went and looked at those three sites&rsquo; robots.txt files, expecting to find the problem there. Instead I found this:

# AI assistants & answer engines — explicitly welcomed for discoverability<br>User-agent: GPTBot<br>User-agent: OAI-SearchBot<br>User-agent: ChatGPT-User<br>User-agent: ClaudeBot<br>User-agent: PerplexityBot<br>User-agent: Google-Extended<br>User-agent: CCBot<br>Allow: /

Immaculate. All three also serve a valid llms.txt. All three have canonical tags, Open Graph images, Twitter cards, and JSON-LD structured data in the .

I had rolled out a red carpet, printed a welcome sign in seven languages, and pointed it all at an empty room.

The eight sites that scored fine are not better marketed. They are just built on frameworks that render on the server. That is the entire difference. Two years of work on a product, and its discoverability came down to a build-time decision I made in an afternoon.

Why now

Why this matters more this year than last

You could reasonably have ignored this in 2024. You cannot now, for three reasons that are all measurable.

One: ranking and being cited have come apart. Seer Interactive analysed 5,000+ URLs across ChatGPT, Perplexity and AI Overviews and found the overlap between top Google...