Most Googlebots are fake — Digital Seams

Open Menu<br>Close Menu

Digital Seams

Open Menu<br>Close Menu

Most Googlebots are fake

Jul 26

Written By Bobbie Chen

I recently saw complaints about Google’s search engine crawler taking down websites by sending them too much traffic:<br>We had googlebot blast a random customer system and almost cause an outage, this is when I first learnt that google will use it for AI training also. It's honestly kind of frustrating also because you then search on it and theres (was) nothing on how you are meant to "correctly" tell google to fuck off, and not use it like that.<br>— User jofzar on Hacker News<br>Google's web scraping functionality has been acting as a ddos for more than two decades. I've seen literally hundreds of reports of them attacking websites and taking them down, where there's nothing you can do but accept the traffic, or get delisted<br>— User 20k on Hacker News

But I think these complaints are misguided. This traffic doesn’t come from Google at all. It comes from other people’s bots impersonating Googlebot’s name for misleading and arguably-abusive purposes.<br>Most* Googlebots are fake.<br>The user agent header is a pinky promise<br>When people say that they see traffic from Googlebot, they usually mean that they see Googlebot in the user agent header, usually embedded in a string like this:<br>User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Or, if you’re looking at nginx-style server logs, it might look something like this:<br>78.142.228.252 - - [17/Jul/2026:14:57:44 +0000] "GET /blog HTTP/1.1" 200 4821 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html )"

This is all fine and dandy, but this header doesn’t mean it’s the real Googlebot at all. The user agent is a voluntary self-identification. Anyone making an HTTP request can set that user agent value to anything!

Peter Steiner for The New Yorker, via Wikipedia

On the internet, no one knows you’re a dog… and the flip side is that I can claim to be a dog (or a Googlebot) whenever I feel like it.<br>These liars impersonate Googlebot in hopes that it will help them bypass bot controls; indeed, if you look up that IP in my log above, it comes from Virtual Machine Solutions LLC. This is a hosting provider, so someone (not Google) rented a server from them to send this traffic.<br>People do this, all the time, and it seems to have hit a new scale recently. Chris Siebenmann theorizes that “it’s a large scale campaign by a single abusive crawler by people who can afford to obtain a lot of servers at a lot of different hosting providers”, rather than a widespread rise in popularity for an old trick.<br>Lightning Q&A<br>Q: How do I verify that a request came from the real Googlebot?<br>A: Here are Google’s docs for verifying Googlebot; in short, you can spot-check with a couple of `host` commands or check against Google’s published IP ranges. Most major crawlers publish some kind of IP list, but the details vary; the JAFAR proposal aims to standardize these lists, though it’s still working its way through the standard-making process.<br>Q: What if I looked up the IP address and saw it was registered to Google LLC?<br>A: I’d still run through the verification process, since “Google LLC” is also the registration for Google Cloud hosting IP addresses. Though personally, I wasn’t able to find any recent examples of this - I wouldn’t be surprised if the Google Cloud anti-abuse team had some kind of controls to crack down on this.<br>Q: I checked and it really is real Googlebot. What should I do?<br>A: See Google’s docs on how to handle overcrawling of your site.<br>Q: You mean to tell me people can lie on the internet?<br>A: Yes :(<br>Q: What if we added another field to stop them from lying?<br>A: Please refer to RFC 3514, the “evil bit” standard. Unfortunately, bad actors do not comply with this standard.<br>On a more serious note, I’ve written at length about the Web Bot Auth HTTP Signatures proposal for my employer (as always, opinions on this blog my own). That proposal is a little more complicated than just another field since it involves a cryptographic signature, but I think it’s a reasonable path to “a user agent you can trust”.<br>*Q: You’ve convinced me some Googlebots are fake. But is it really “most”?<br>A: Okay, I concede that I don’t have data to prove this. I offer this weaker claim: most stories of disruptive Googlebot traffic (like the quotes at the start of this post) are caused by fake Googlebots. In my experience, the real Googlebot is very well-behaved; meanwhile, fake Googlebots are happy to hammer on a site until it explodes.<br>That being said, if you have seen disruptive behavior from Googlebots (real or fake), I’d like to hear about it - reach out at bobbie@digitalseams.com.

Bobbie Chen

Next

Next

Zero-defects code: the prescient Microsoft memo from 1989

Email Address

Subscribe

Thank you! You’ll receive an email, please click the link to confirm.

Digital Seams<br>Piecing together how software...