Gandalf - Botblocker made by NerdVPN.de

🔮What is Gandalf?

Gandalf is NerdVPN.de's bot protection system, a self-hosted, privacy-first access guard that blocks automated bots, scrapers, crawlers, and AI agents from accessing its services.

When you visit a page protected by Gandalf, he takes a brief look at your browser and decides<br>whether you're a legitimate visitor or a suspicious automated bot. Legitimate users on supported browsers<br>experience only a brief, unobtrusive verification step. Suspicious or anomalous requests are subject<br>to an additional challenge or are blocked outright.

🧙<br>"You cannot pass. I am a servant of the Secret Fire, wielder of the flame of Anor. The dark fire will not avail you, flame of Udûn. Go back to the Shadow. You shall not pass!"<br>— Gandalf, The Lord of the Rings, Book II, Chapter 5: "The Bridge of Khazad-dûm"

Having trouble? If you are on a supported browser and still can't get through,<br>check the troubleshooting tips below. If you believe you have been blocked incorrectly,<br>please report it.

⚔️Why Bot Protection?

The modern web has a bot problem and it's getting worse.

According to Cloudflare Radar and the CrowdSec Threat Intelligence Network, nearly 45% of all HTTP traffic on the internet today is generated by automated clients rather than human users, a share that has been growing exponentially, driven by the rise of AI-powered crawlers and scrapers. For independent and FOSS services, the picture is often far worse than the internet-wide average. Many sysadmins report bot shares of up to 95% of all incoming requests. NerdVPN.de is no exception: the share of blocked bot traffic sits at around 90% , amounting to roughly 6 million requests per day .<br>The majority of this traffic is harmful: it includes scrapers harvesting content without permission, AI crawlers consuming bandwidth to feed training datasets, credential-stuffing bots probing for weak passwords, and spam harvesters collecting contact information for abuse.

For small and independent operators, this is not an abstract threat. Unmitigated bot traffic means real server costs, degraded performance for legitimate users, and loss of control over your own content.

Beyond automated bots and scrapers, there is a growing number of bad actors who deliberately exploit publicly accessible service instances as free infrastructure for their own applications. Rather than running their own instance or using an official API, they silently piggyback on community-operated services, consuming resources, bypassing rate limits, and degrading the experience for everyone else. This kind of abuse is especially common with alternative frontends, where bad actors route their app's traffic through public instances to avoid the costs and restrictions of the original platform's API.

The standard and, in my opinion, lazy answer to this problem is to outsource it, to Cloudflare, Google reCAPTCHA, or similar third-party services. But the trade-off is significant: it means routing all visitor traffic through infrastructure you don't control, handing sensitive data to US-based corporations, and accepting their terms, their outages, their business models, and their history of misusing and selling user data to the highest bidder and every government that asks them to.

Gandalf exists because that trade-off is unacceptable.

Bot protection should not require surrendering your users' privacy!

The cost of doing nothing

NerdVPN.de operates several privacy-friendly alternative frontends, including Invidious (a YouTube frontend) and Redlib (a Reddit frontend). These services proxy requests to their respective upstream platforms and those platforms actively rate-limit or block instances that generate too many requests in a short period of time.

Without bot protection, unchecked automated traffic would exhaust these upstream quotas within minutes (yes, really), rendering the services effectively unusable for real users. A botblocker ensures that the available capacity is reserved for humans.

A necessary trade-off

As a consequence of this protection, certain endpoints, including APIs, RSS feeds, and other programmatic access points, are also blocked. Additionally, link preview generation in messengers and social platforms (e.g. WhatsApp, Telegram, Signal) will not work: the automated crawlers these services use to generate previews are indistinguishable from bots and are consequently intercepted as such. This is an unfortunate but unavoidable side effect: there is no reliable way to distinguish a legitimate script or crawler from an abusive bot at scale without blocking both. Programmatic access to NerdVPN.de services is against the terms of use and will be blocked, a standard browser remains the only supported client.

A grim future

Bot detection is getting harder. Modern bots are no longer crude scripts firing raw HTTP requests; they run full browser engines, solve JavaScript challenges, mimic human interaction patterns, and in many cases...