AI crawlers now match Googlebot in traffic to our site | Mojo Dojo | Mojo DojoSkip to main content
AI crawlers now match Googlebot in traffic to our site<br>Ajay C<br>·17 June 2026·1 min read
What traffic & bots flow to your site after you make it to the front page of Hacker News? Do AI bots crawl as much as Google? Which is the most aggressive bot? Which is the least and thriftiest?<br>A recent post hit the front page on hacker news. Google just made you a search quality rater.You won't get paid.<br>This was something I wished to know so I built a little bot detector on this site. Of course using Claude, the bot.<br>The code basically measured the traffic over 30 days and I built a simple analyser within the code to output the following data.<br>Across 240,060 visits from 24 distinct bots, AI crawlers (35.0% of traffic) tied search engines (35.0%) for the top slot.<br>If you care about this industry then that's the headline. The rest is nerding in data.<br>The setup<br>We log every visit from a recognised bot user-agent into a single table. The fields are bot name, category, path, day, hit count. 240,060 hits, 24 bots, 5,074 distinct URLs touched.<br>We bucket bots into five categories: search (Google, Bing, DuckDuckGo, etc.), ai (GPTBot, ClaudeBot, PerplexityBot, AmazonBot, etc.), seo (Ahrefs, Semrush, Majestic, Screaming Frog), social (LinkedIn, Facebook, Twitter, Pinterest), and archive (Internet Archive).<br>Category split<br>AI crawlers tied search engines at 35.0% each.<br>Category<br>Hits<br>Share<br>AI crawlers<br>84,080<br>35.0%<br>Search engines<br>83,980<br>35.0%<br>SEO tools<br>50,820<br>21.2%<br>Social previews<br>20,700<br>8.6%<br>Archive<br>480<br>0.2%<br>Most aggressive bots<br>Hits per day, averaged over the period.<br>Bot<br>Category<br>Hits/day<br>Total hits<br>Unique paths<br>Googlebot<br>search<br>1,228<br>36,840<br>1,474<br>Bingbot<br>search<br>1,187<br>35,610<br>1,785<br>AmazonBot<br>ai<br>1,101<br>33,040<br>1,442<br>MajesticBot<br>seo<br>1,062<br>31,860<br>2,026<br>ChatGPT-User<br>ai<br>812<br>24,350<br>239<br>ClaudeBot<br>ai<br>581<br>17,430<br>647<br>AhrefsBot<br>seo<br>485<br>14,550<br>1,279<br>LinkedInBot<br>social<br>478<br>14,340<br>15<br>Most polite bots<br>Hits per path. Lower = visit and leave.<br>Bot<br>Category<br>Hits/path<br>Unique paths<br>AhrefsBot<br>seo<br>11.4<br>1,279<br>Applebot<br>search<br>11.9<br>575<br>MajesticBot<br>seo<br>15.7<br>2,026<br>SemrushBot<br>seo<br>16.2<br>265<br>Bingbot<br>search<br>20.0<br>1,785<br>AmazonBot<br>ai<br>22.9<br>1,442<br>Googlebot<br>search<br>25.0<br>1,474<br>ClaudeBot<br>ai<br>26.9<br>647<br>Most obsessive bots<br>Re-scrapers. Few URLs, hammered repeatedly.<br>Bot<br>Hits/path<br>Total hits<br>Distinct paths<br>LinkedInBot<br>956.0<br>14,340<br>15<br>FacebookBot<br>170.3<br>6,130<br>36<br>ChatGPT-User<br>101.9<br>24,350<br>239<br>Internet Archive<br>16.0<br>480<br>30<br>Most thorough explorer<br>Distinct URLs touched.<br>Bot<br>Distinct paths visited<br>Total hits<br>MajesticBot<br>2,026<br>31,860<br>Bingbot<br>1,785<br>35,610<br>Googlebot<br>1,474<br>36,840<br>AmazonBot<br>1,442<br>33,040<br>AhrefsBot<br>1,279<br>14,550<br>ClaudeBot<br>647<br>17,430<br>Applebot<br>575<br>6,840<br>Rare visitors<br>Bots that barely show up across 30 days.<br>Bot<br>Total hits<br>Days active<br>Internet Archive<br>480<br>Baiduspider<br>480<br>DuckDuckBot<br>470<br>11<br>Twitterbot<br>160<br>DotBot<br>80<br>Pinterestbot<br>70<br>Google-Extended (Gemini)<br>60<br>Screaming Frog<br>50<br>Methodology<br>Source: nexus_bot_visits table on mojodojo.io's production database. One row per (bot, path, day) with a hit counter.
Bot detection: User-Agent substring match against a curated list of 32 known bot patterns. Unknown / spoofed UAs are excluded.
From our research<br>State of B2B AI Visibility 2026<br>We audited 712 B2B companies. 89% are invisible in AI search. Industry benchmarks, the 4 patterns of leaders, and a 30-day fix list.<br>Read the report →<br>Filed under<br>ai-visibilityseoanalyticsbots
Written by<br>Ajay C<br>Ajay is the CTO of Mojo Dojo with over 15 years of experience in digital marketing.
Keep reading<br>More from Insights
Google just made you a search quality rater. You won't get paid.<br>TL;DR AI is killing the click. Death of the click kills the incentive for links. This leaves Google...<br>7 June 2026
Meta is selling organic reach for a flat fee.<br>For fifteen years, agencies sold clients a dream: "earned media" on someone else's property. We...<br>7 June 2026
How to write a great value proposition? (Template included)<br>What is value proposition? A great value proposition makes your business the obvious choice amongst...<br>6 June 2026
View all articles