“Stealth Crawlers” Are Not a Threat to the Open Web. Bills Targeting Them Would Be. | Electronic Frontier Foundation

Skip to main content

AboutContact

Press

People

Opportunities

IssuesFree Speech

Privacy

Creativity and Innovation

Transparency

International

Security

Artificial Intelligence

Our WorkDeeplinks Blog

Press Releases

Events

Legal Cases

Whitepapers

Podcasts

Annual Reports

Take ActionAction Center

Volunteer

Follow EFF

ToolsPrivacy Badger

Surveillance Self-Defense

Certbot

Atlas of Surveillance

Cover Your Tracks

Street Level Surveillance

apkeep

Shop

DonateDonate to EFF

Shop

Giving Societies

Sponsorships

Other Ways to Give

Membership FAQ

Email updates on news, actions,

and events in your area.

Join EFF Lists

Copyright (CC BY)

Trademark

Privacy Policy

Thanks

Electronic Frontier Foundation

Donate

If you use technology, this fight is yours.Donate today

EFFecting Change: If You Own It, Why Can't You Fix It? on July 23

“Stealth Crawlers” Are Not a Threat to the Open Web. Bills Targeting Them Would Be.

DEEPLINKS BLOG

By Tori Noble<br>July 20, 2026

“Stealth Crawlers” Are Not a Threat to the Open Web. Bills Targeting Them Would Be.

Share It

Share on Mastodon<br>Share on Bluesky<br>Share on Facebook<br>Copy link

There’s a new boogeyman in the battles over AI: so-called “stealth crawlers.” We’ll admit it—the term “stealth crawlers” sounds quite nefarious. In reality, they’re anything but.

“Stealth crawlers” are simply automated tools to access and collect public web data—without disclosing the user’s identity. Private crawlers like these facilitate all kinds of important work that benefits the public, including investigative reporting, academic research, cybersecurity protection, and more.

Many publishers want to unmask crawlers anyways—and are pushing for new legislation that would give them new powers to do so. These legislative proposals threaten the open web, user privacy, and valuable research without directly addressing the problems they’re supposedly intending to solve.

Alarmingly, these harmful proposals are gaining traction. The New York state legislature has already passed such a bill, the NY Stealth Crawler Protection Act, which is now on Governor Hochul’s desk. We expect to see similar bills introduced in other states, and potentially in Congress. That’s a big problem for the open web—and the many benefits it provides.

Anonymous crawling is worth protecting

Anonymous crawling enables some of the most publicly beneficial uses of the open web. Researchers, journalists, and other watchdog groups use unidentified automated tools to gather the information necessary to hold powerful institutions accountable and protect the public.

Anonymous crawling fuels important investigative journalism. For example, The Markup, a non-profit news site, used anonymous crawlers to investigate potentially anti-competitive practices by tech companies, such as Amazon’s tendency to prioritize Amazon brands and Amazon-exclusive products over competitors with higher ratings. The crawlers identified themselves as ordinary Firefox browsers to web servers, which allowed The Markup to understand how Amazon search results pages would appear to ordinary users. Similarly, ProPublica used an automated tool designed to simulate an ordinary Amazon customer to reveal that the site steered shoppers to more expensive products over cheaper alternatives.

Anonymous web scraping is also crucial for cybersecurity professionals, who use automated tools to monitor the web for information that helps them protect against malicious attackers. Privacy tools, including EFF’s own Privacy Badger, also crawl sites anonymously to identify trackers without compromising user privacy.

However, without the ability to scrape anonymously, these tools would likely be blocked. Sites can—and do—block crawlers operated by researchers, journalists, and activists who criticize them. For example, Facebook shut down accounts belonging to researchers who used automated tools to study misinformation on the platform and demanded that they take down published research. Many sites block automated access by anyone who hasn’t paid to crawl public webpages.

Unmasking crawlers threatens the open web

News publishers—and their allies in government—say that unmasking crawlers is necessary to protect news organizations from technological strain caused by AI-related crawling, and fears that AI could reduce news sites’ traffic and ad revenue. These are legitimate concerns.

But enacting broad, reactionary restrictions on automated access is not the answer. Legislation targeting anonymous crawling threatens the open web, user privacy, and valuable research without actually addressing these technological and potential economic harms of scraping.

The New York state legislature recently passed the NY Stealth Crawler Protection Act, a law that would make it illegal to crawl news websites without revealing who is operating the crawler...