Responsible bot operation
-->
Responsible bot operation
Author : Aaron P. MacSween
Published : 2025-12-01
I've written two articles about poorly behaved web-crawling<br>robots recently,<br>first about a somewhat novel method to distinguish bots<br>from legitimate human traffic<br>("AI scrapers request commented scripts"),<br>then another unpacking some of the responses to that article<br>("Algorithmic sabotage debrief").<br>Having already spent some time outlining the various things an<br>operator can do to earn a sysadmin's malice<br>I figured it might be worthwhile to talk at least briefly<br>about how to operate a bot responsibly.
Most of the practices I intend to describe are already<br>in use by a variety of web-crawlers, however,<br>I haven't been able to find a formal name to describe<br>them all, nor have I come across any articles<br>which cover all the behaviours that I consider important.<br>I'll link to examples where they are available,<br>and include some concrete measurements of behaviour I've<br>observed on my own servers wherever possible.
Robots.txt<br>If you're reading this then it's very likely you've already<br>heard of robots.txt before (see robotstxt.org otherwise).<br>Basically, it's a simple text file that a website can host<br>which serves as a means of communicating to various bots<br>how the site expects them to behave.<br>Particular bots can be allowed or disallowed across the whole site,<br>or with regard to particular resources, including paths and general<br>patterns like file extensions.
The format was introduced in 1994, but somehow was not formally described<br>until 2022 (in RFC 9309: Robots Exclusion Protocol, drafted primarily by Google employees).<br>Even then, common extensions like Sitemaps<br>(another protocol designed by Google employees)<br>were not explicitly defined in that formal description<br>except for a note that additional fields may be supported:
Crawlers MAY interpret other records that are not part of the robots.txt protocol -- for example, "Sitemaps" [SITEMAPS]. Crawlers MAY be lenient when interpreting other records. For example, crawlers may accept common misspellings of the record.
Many crawlers support the Crawl-Delay, which specifies a number<br>of seconds to wait between requests, but for some reason it was not<br>included in the specification.<br>I'm not aware of any commonly agreed upon upper bound for this delay,<br>and because it's so poorly specified it can be hard to validate how<br>well a particular crawler adheres to the protocol.<br>Some others expect a Request-rate<br>(Seznam.cz for instance),<br>so in practice you might have to include both directives<br>with their own particular syntax.
Another issue to be aware of is that there are various crawlers<br>that will interpret robots.txt files somewhat creatively.<br>Google's dominance over search is so dramatic that many site administrators<br>will include directives for them and not bother with others.<br>This has led to the following policy from applebot:
If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions.
...and this similar policy from "ImageSift":
If there is no rule targeting ImagesiftBot, but there is a rule targeting Googlebot, then ImagesiftBot will follow the Googlebot directives.
From what I've seen both of these bots still obey catch-all directives<br>like the following:
User-agent: *<br>Disallow: /
...though this was not particularly clear based on how their policies were written.
So, for anyone looking to get into responsible web-crawling,<br>I would recommend the following:
Read RFC 9309.
Look into frameworks for crawling that implement support for the usual extensions and have been tested against a variety of edge cases. I have no direct experience with this project, but Ethicrawl seems like a somewhat competent reference implementation. I've also heard good things about BeautifulSoup.
Don't get creative in how you interpret a site's robots.txt file. Stick to the usual definitions, and if your crawler is disallowed by a site, then don't crawl.
User-agent strings<br>A User-Agent header (or UA string )<br>is a bit of text that browsers and bots alike can send as part of a request<br>to identify themselves.<br>It's entirely voluntary, and a bad bot can send a fake string<br>to pretend to be something else (more on that later),<br>but it's a good starting point.
Wikipedia's user-agent policy<br>is worth reading in its entirety, but I'll quote the most relevant bits:
As of February 15, 2010, Wikimedia sites require a HTTP User-Agent header for all requests. This was an operative decision made by the technical staff and was announced and discussed on the technical mailing list. The rationale is, that clients that do not send a User-Agent string are mostly ill behaved scripts that cause a lot of load on the servers, without benefiting the projects. User-Agent strings that begin with non-descriptive default values, such as python-requests/x, may also be blocked from Wikimedia sites (or parts of a website, e.g. api.php).
I implement a similar...