Fighting Bot Traffic · Markus Tenghamn

Over the last week I have received a 900% increase in active users and a 300% increase in page views on one of my other websites according to Google Analytics. Here I list some of my thoughts around this issue, potential solutions and what I ultimately decided to do.

Bots From Singapore

I run a community website around gaming and although it is normal to see traffic from all over the world it was unusual to see such high amounts of traffic from both China and Singapore. You might think this is AI related and it very well could be but these bots are not accessing the website in a friendly or respectful manner. The pages are fairly static with a few new user submitted posts every day so there is no real need to access the website at this rate, you would be better of caching the page data.

Now, is there a reason to block these bots? For me personally it could affect my costs as I serve images from Backblaze and the cost there really has no limit. The more images that are served from the website the more my costs go up even if they are low as I do a lot of caching and things to reduce costs. So this was my first reason for wanting to block these bots.

Another reason why I wanted to block these bots is that it really skews my statistics for the website. How can I understand my Google analytics if 70% of traffic is bots that mess with the data? I do also run some ads to support the website and all of the metrics that determine ad revenue are also affected by the bots.

With a basic understanding of the issue I could now set out to block these AI bots. I also felt that it was really important to try to avoid blocking any real users and this is why I did not resort to simply blocking entire countries.

Super Bot Fight Mode Doesn't Always Work

I do use Cloudflare for my website and it is really great for caching. Remember how I said that Backblaze is a cost for serving images on the site? Well, Cloudflare caches about 90% of the content server on this website every month, or about 7TB of data served from cache. Cloudflare also offers another solution called Super Bot Fight Mode which I could activate at the click of a button for the entire website. I do need a Pro subscription in order to use this which is about 25 USD per month.

Super Bot Fight Mode is really great as I instantly saw results and a reduction in bot requests. Suspected bots would get blocked with the Cloudflare captcha challenge and only be let through to my website if they could solve it. I did receive a couple of reports from users, especially from Russia for some reason, that they were presented with the challenge screen but I figured as long as they could get through and as long as most users were not affected then this was a really good outcome for blocking bot traffic.

One thing I had not considered was my API which is open and free to use. By turning this bot fighting mode on I had ended up blocking all integrations talking to the API. This was solved with a security rule similar to the following:

http.request.method eq "GET"<br>and len(http.request.headers["x-api-key"]) > 0<br>and starts_with(http.request.uri.path, "/api/")<br>and not (<br>ends_with(http.request.uri.path, "/download")<br>or starts_with(http.request.uri.path, "/api/download")<br>or starts_with(http.request.uri.path, "/api/files/")

This rule allows access to most API endpoints but disallows direct file downloads as this is not part of the public API documentation. You can also view how this security rule is performing in Cloudflare and see how many skip actions were taken.

At first I thought I had solved the problem, Cloudflare would take care of everything for me and I just had to pay for a PRO subscription, that's it. However, as time went on I noticed the amount of traffic from Singapore and China increasing yet again according to Google Analytics. It was time to do some more research.

800×600 Screen Resolution

I added some of my own logging on the website with user agent, screen resolution, cloudflare country header and more. I wanted to see what my traffic was like in the hopes that I could filter out the remaining bot traffic in some way. The problematic users were those that looked real, loaded JS and were not blocked by Cloudflare.

Resolution<br>Hits<br>Share of JS views

800×600<br>402,903<br>72%

1600×1600<br>12,600<br>2.3%

1024×1024<br>4,904<br>0.9%

After looking at the data it seemed like screen resulution is a pretty good indicator. Most of these resolutions also had the same or a very similar user agent made to look like a normal browser, like this: ´Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)´. So this gave me a way to detect the agents but I couldn't be 100% sure that these were all 100% bots.

I am not sure why but Google analytics actually showed different resolutions from my own javascript code but still pointed to square and 800x600 as being bots based on average engagement time.

I spent a bit of time Googling and found that the 800x600...