An update on the scraper situation [LWN.net]
LWN<br>.net<br>News from the source
Content Weekly Edition<br>Archives<br>Search<br>Kernel<br>Security<br>Events calendar<br>Unread comments
LWN FAQ<br>Write for us
User:<br>Password: |
Log in /<br>Subscribe /<br>Register
An update on the scraper situation
[LWN subscriber-only content]
LWN needs you
LWN counts on its subscribing readers to support its mission of creating relevant human-written news coverage of the Linux and free-software communities. Please subscribe to LWN and help to keep us on the net.
As a special offer, subscribe to LWN now for at least six months, and receive a 25% discount on your subscription.
[+] Proceed to the article
By Jonathan Corbet<br>July 10, 2026
Our article "Fighting the AI scraper bot<br>scourge", published in early 2025, discussed the problem of widespread<br>scraping of web sites in search of training data for large language models<br>and related projects. This activity overwhelms sites with traffic. Over a<br>year after that article is published, the problem is still growing. The<br>hammering of sites by shadowy actors has reached new heights, and the open<br>web is becoming increasingly difficult to maintain. Where is this traffic<br>coming from, and what can be done about it?
Residential proxies
As was described last year, scraper attacks come from a huge number of<br>sources across the net. It is not unusual to see coordinated requests from<br>millions of unique IP addresses over the course of a few hours, each of<br>which hits the site at most two or three times. Attacker-controlled data,<br>such as the user-agent field, is entirely fictional; each hit is meant to<br>look like just another human with a web browser. There are ways to tell<br>the difference — the bots usually do not fetch images or CSS, for example —<br>but, by the time that determination is made, the address in question will<br>not be used again. Blocking the address at that point is just a waste of<br>time.
This traffic comes predominantly from residential and mobile networks,<br>directed by central command-and-control nodes. Software is installed on<br>ordinary systems that takes orders from a control node, fetches web pages<br>on demand, and forwards the resulting data back to the controller. Much of<br>the time, this activity occurs without the knowledge or consent of the<br>owner of the device in question. The term "residential proxies" is used to<br>describe systems that are used in this way.
There are a few different (on the surface, at least) types of operator<br>running residential-proxy networks to attack web sites. One type is purely<br>criminal, running scrapers on systems that have been compromised with some<br>sort of malware. At the beginning of the year, Google acted<br>to take down a bot network called IPIDEA and provided a lot of<br>information about how these operations work. The shutdown of IPIDEA<br>correlated with a significant reduction in scraper traffic here at LWN;<br>things were relatively peaceful for a few months. That period of peace has<br>since come to an end, though.
More recently, media-streaming devices have been identified<br>as a major carrier of malicious scraping software. Sometimes the devices<br>are compromised at the source; other times, they are just poorly secured<br>and easily compromised after the fact.
The second sort of operator works more overtly, pretending to a degree of<br>legitimacy and offering "ethically sourced" IP addresses. A company called<br>Bright Data is one of the most prominent of these; it happily advertises<br>its prowess at getting around web-site access controls and traffic limits.<br>Bright Data offers a "free" VPN service; all that is needed is for the user<br>to give Bright Data the ability to route traffic through the user's device<br>— to become a part of the company's residential-proxy network, in other<br>words. Every phone or other device that makes use of this VPN becomes yet<br>another endpoint that will be used to attack web sites.
There are many other examples of this type of operator out there; often<br>they offer a library that app developers can link into their offerings and<br>be paid for hijacking their users' network connections. One of them even<br>sent us a query about running an ad for its SDK on LWN; that was, it<br>suffices to say, a short conversation. In general, these companies range<br>from those that aspire toward some appearance of legitimacy, advertising<br>"GDPR compliance" for example, to others that are just overtly sleazy.
While these residential-proxy networks are used for web-site scraping, it<br>is worth emphasizing that these operators have the ability to run code that<br>accesses resources on whatever networks millions of devices happen to be<br>connected to. To assume that this type of access would only be used for<br>scraping would be naive at best.
Then, of course, there are the high-profile companies developing models as<br>their core business. These companies do their own scraping; the traffic<br>that can be easily attributed to them is clearly identified in the<br>user-agent field and, as a general rule, observes...