GitHub - MerqryLabs/ai-crawler-visibility · GitHub

/" data-turbo-transient="true" />

Skip to content

Search or jump to...

Search code, repositories, users, issues, pull requests...

-->

Search

Clear

Search syntax tips

Provide feedback

--><br>We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Cancel

Submit feedback

Saved searches

Use saved searches to filter your results more quickly

-->

Name

Query

To see all available qualifiers, see our documentation.

Cancel

Create saved search

Sign in

/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up

Appearance settings

Resetting focus

You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

{{ message }}

MerqryLabs

ai-crawler-visibility

Public

Notifications<br>You must be signed in to change notification settings

Fork

Star

main

BranchesTags

Go to file

CodeOpen more actions menu

Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit

History<br>1 Commit<br>1 Commit

README.md

README.md

ai-crawler-visibility.zip

ai-crawler-visibility.zip

View all files

Repository files navigation

AI Crawler Visibility

A Claude Code skill that tells you whether your website is actually visible to AI search — and exactly how to fix it if it isn't.

ChatGPT Search, Claude, and Perplexity are sending people to websites now. But the crawlers that feed them — ClaudeBot, GPTBot, PerplexityBot, Bytespider, Meta-ExternalAgent — do not run JavaScript. They read only the raw HTML your server returns.

Google's renderer has handled JavaScript for years, so a client-rendered site (React, Vite, Vue, Svelte, any SPA) can rank perfectly fine on Google and still be completely invisible in AI answers .

If you've ever thought "we get Google traffic but nothing from AI" — or you shipped a slick vibe-coded SPA and you're nowhere in ChatGPT — this is almost always why.

What it does

This skill fetches your page(s) the same way a non-JS crawler does — raw HTML, no browser — and reports, per page, whether your real content and key signals (title, meta description, canonical, Open Graph, JSON-LD, crawlable links) are actually present.

Then it prescribes a concrete, framework-specific fix . Not "consider SSR" — the exact tool, config change, and one-line way to verify it worked.

Every page gets one of three verdicts:

Verdict<br>What it means

✅ VISIBLE<br>Real content is in the raw HTML. The crawler can read you.

⚠️ PARTIAL<br>Only a fraction is in raw HTML; the main body is JS-rendered.

❌ INVISIBLE<br>The crawler gets an empty shell (). Nothing useful is there.

It diagnoses and prescribes . It does not write marketing copy, do keyword research, or chase backlinks.

Install

This is a Claude Code Agent Skill. Drop the folder into your skills directory and Claude Code picks it up automatically.

Personal (available in every project):

/ai-crawler-visibility.git \<br>~/.claude/skills/ai-crawler-visibility">git clone https://github.com/merqrylabs>/ai-crawler-visibility.git \<br>~/.claude/skills/ai-crawler-visibility

Project-scoped (just this repo):

/ai-crawler-visibility.git \<br>./.claude/skills/ai-crawler-visibility">git clone https://github.com/merqrylabs>/ai-crawler-visibility.git \<br>./.claude/skills/ai-crawler-visibility

That's it. No npm install, no API keys, no third-party packages — the analyzer runs on plain Python 3.

Check the official Agent Skills docs for the canonical skills path if your setup differs.

Use it

Just talk to Claude Code in natural language. The skill triggers on its own when you ask the kind of question it's built for:

"Why isn't my site showing up in ChatGPT?"

"Is my new site visible to AI search engines?"

"I added JSON-LD but rich results still aren't showing."

You can also run the bundled analyzer directly:

# Single page<br>python3 scripts/check_visibility.py https://thesite.com

# Several specific pages<br>python3 scripts/check_visibility.py https://thesite.com https://thesite.com/pricing

# Sample from a sitemap (checks up to --limit pages)<br>python3 scripts/check_visibility.py --sitemap https://thesite.com/sitemap.xml --limit 10

Add --json to process results programmatically.

⚠️ Always check your production URL, not localhost — local dev servers don't reflect how the deployed site is served to crawlers.

What the report looks like

# AI Crawler Visibility Report

## Verdict<br>https://thesite.com — INVISIBLE<br>https://thesite.com/pricing — INVISIBLE

## What the crawler actually sees<br>/ : 0 visible words. Missing from raw HTML: meta description,<br>canonical, Open Graph, JSON-LD, crawlable links.

## Fixes (highest impact first)<br>[Concrete, framework-specific steps — each with how to verify]

## How to confirm the fix<br>View Page Source (not Inspect) and confirm your headline text<br>and JSON-LD appear...