Blocking AI Crawlers on your Mastodon Server, Nginx Edition - Justin Ribeiro, Ph.D.
There are few things worse than getting an alert notification from a server at night. Your mind races—is my redundancy working, should I grab a fire extinguisher—knowing that the glaring light of your SSH session in bed will wake your spouse. Then you’ll have two problems.
Such an event found me the other day, for of all things, my single no-frills instance that runs my Mastodon server. This is a tiny server with few resources because it doesn’t need them; I am the only daily active user with other family using it sparingly. Load is not an issue, but on this night, something was making the network very very angry.
It was AI, because of course it was.
Specifically, the AI crawlers just hammer and consume without reason. I’ve noted this before, though this is the first time I’ve found them roaming around my Mastodon server.
To give myself a fighting chance, I quickly put an nginx include into Mastodon’s config to block the aforementioned AI crawlers. I based mine on nginx-block-ai-bots.conf, which I also use as a front as part of my other infra.
There’s no specific documentation on how to do this on the Mastodon docs—why would there be?—but for those seeking refuge hopefully this helps:
On your Mastodon server either clone ai-robots-txt/ai.robots.txt repo so you have the related nginx-block-ai-bots.conf or write a curl script to grab it from a cron job on some weekly basis.
If you followed the nginx install docs for Mastodon, then you likely have a /etc/nginx/sites-available/mastodon or some related nginx config file (mine is name based on the domain name because that’s how I config virtual hosts, so your mileage will vary, but it must be there somewhere).
In this file, there is a server block that looks something like this:
server {<br>listen 443 ssl http2;<br>listen [::]:443 ssl http2;<br>server_name example.com;
ssl_protocols TLSv1.2 TLSv1.3;
# blah blah blah
In that server block write the include to the nginx-block-ai-bots.conf:
server {<br># block as many AI crawlers as possible so I don't owe $$$$<br>include /your/path/here/nginx-block-ai-bots.conf;
listen 443 ssl http2;<br>listen [::]:443 ssl http2;<br>server_name example.com;
ssl_protocols TLSv1.2 TLSv1.3;
# blah blah blah
Reload nginx, ala systemctl reload nginx.server or whatever your setup is on your server.
That’s it. For an added bonus, you can also update the robots.txt in ${where-ever-mastodon-is}/public/robots.txt to add the list from ai.robots.txt/robots.txt.
This stemmed the tide the other night and I was not again woken from my slumber. It is an ever increasing problem—largely a chase I will eventually lose—to be fighting the tide of bad crawlers with bad rules that are run by inexperienced, dismissive, and downright exploitive corporations or start-ups.
You don’t scan the web this way you hacks. So shields up.
Justin Ribeiro, Ph.D.<br>Dr.<br>Justin<br>D.<br>Ribeiro<br>Ph.D.
I ♥ the open web. Film photographer and mediocre darkroom printer. Maker of sometimes interesting things. Coder of open source curiosities.<br>Researcher of creativity. Always exploring.
Web site<br>Mastodon<br>Codeberg<br>Github<br>YouTube<br>justin@justinribeiro.com<br>justin.ribeiro@case.edu<br>justin.ribeiro@stevens.edu