Cloudflare blocks AI agents by default | Blog<br>Back<br>July 10, 2026<br>Cloudflare blocks AI agents by default now<br>Your agent reads the block page and reports it back as fact. Here’s how often that happens, and how to fix it.<br>GitHub

My agent was confidently wrong<br>As of July 1, 2025, Cloudflare blocks AI crawlers by default on every new domain, and it has since turned away hundreds of billions of bot requests. Most of the coverage was about the big fight, AI labs against publishers over training data. That wasn’t my problem.<br>My problem was an agent that kept coming back confidently wrong. It would read a source, check a price, fetch a page I asked for, and give me an answer that sounded right and wasn’t. It was reading the block page instead of the real page: Cloudflare’s “Just a moment” screen, summarized back in the same confident voice it uses for everything else.<br>Nothing in the response said the fetch had failed. So I measured how often it happens, and we built something to fix it.

A 403 with a body is worse than a 403 with nothing<br>The reason this is easy to miss is that nothing in the system thinks anything went wrong.<br>When you make a plain fetch(), httpx.get(), or requests.get()to a bot-protected site, you usually don’t get an exception. You get a 403, sometimes even a 200, with a full HTML body attached: Cloudflare’s “Just a moment” screen, DataDome’s “please enable JavaScript,” PerimeterX’s “Press & Hold,” Amazon’s “Continue shopping” wall.<br>To your retry logic, that looks like success; the request finished. To a language model, the body is just text to read, so it reads it. Hand that 27KB block page to a model and ask for the jobs on it, and it will list them for you, none of which exist. The wrong answer is impossible to tell apart from the right one.<br>A loud failure, a timeout or a clean 429 with an empty body, would trigger a retry or a skip and you’d move on. The quiet block page slips through because it looks like content.

Twelve sites your agent can’t read right now<br>On July 10 I sent an ordinary browser request to a set of popular targets to see which ones Cloudflare turns away. Twelve of the ones people ask about most came back with a hard 403:<br>Use caseSites returning a Cloudflare 403HiringIndeed, Glassdoor, ZipRecruiterSales and market researchCrunchbase, Product Hunt, CapterraFreelanceUpwork, FiverrCommerce and consumerStockX, DoorDash, Coinbase, Udemy<br>If you are building for recruiting, sales research, lead generation, or shopping, you are already running into this. You probably won’t notice. The agent returns a normal-looking answer, built from a page it never actually read.

The benchmark<br>Eight sites, one for each of the major anti-bot vendors, each hit three ways. Naive is curl with an ordinary desktop Chrome user-agent, roughly what an untuned fetch gets. Fortress is the open-source stealth browser we build. Tilion Cloud is the hosted layer on top of it.<br>SiteAnti-botNaiveFortressTilion CloudIndeedCloudflare403jobsZillowPerimeterX403878 listingsWalmartAkamai307pageStockXCloudflarechallengeGraphQL APIBookingproprietary202app loadedAmazonCloudFront WAF404click-wallEtsyDataDome403DataDomeG2DataDome403DataDome<br>The naive fetch got real content zero times out of eight. It got a summarizable body eight times out of eight. And it got a signal the agent could use to know it had been blocked zero times out of eight. That last number is the one that made me write this. Nothing in the response says “you failed.”<br>The open-source Fortress build gets through five of the eight today: the Cloudflare, CloudFront, and PerimeterX targets. It returns clean structured data from Indeed, Zillow, and StockX. DataDome (Etsy, G2) and Amazon’s click-wall are the three it doesn’t clear, and those are the ones Tilion Cloud is built for.

Same URL, side by side<br>Point both at indeed.com/jobs?q=software+engineer. The naive fetch returns 27KB of “Just a moment. Enable JavaScript and cookies to continue.” Fortress returns the actual listings: Senior Software Developer, TherapyNotes, Remote, $110k to $135k; Principal Backend Engineer (AI), Cotiviti, $217k to $258k, full descriptions and all.<br>Point both at zillow.com/homes/for_sale/San-Francisco. The naive fetch returns a 403, “Access to this page has been denied.” Fortress returns 878 live listings with price, beds and baths, sqft, address, and listing agent.<br>The agent on the naive path raises no error on either. It quietly summarizes the block page and moves on.

Getting through is a ladder<br>It isn’t one trick. Which move you need depends on which system you are up against.<br>A single stealth fetch, waiting patiently through the JS challenge, is enough for Cloudflare, CloudFront, and PerimeterX. That covered Indeed, Zillow, and Walmart. Rendering the whole page to structured markdown picks up the SPA and JS-gated content. And when the good data lives behind the scenes, a recon pass drives the page and watches its private API traffic. On StockX that solved the Cloudflare challenge...