How Much Is Crawling Your Content Worth to an AI Bot? | Yale Insights
Skip to main content
Contact Us
Yale University
About Yale Insights
Privacy Policy
Yale SOM
Accessibility
Ideas from the Yale School of Management
Soheil Ghili
Assistant Professor of Marketing
Written by Susie Allen
July 24, 2026
“Best recipe for banana bread.” “How to fix running toilet.” “Is Harry Styles engaged?”<br>For decades, a well-established business model allowed internet users to answer big life questions like these. Search engines directed readers to content on other websites—and those visits helped support content through advertisements, subscriptions, and product sales.<br>AI-assisted search is already threatening the once-symbiotic relationship between search engines and content producers. On Google, for example, many queries now result in an AI Overview. These summaries rely on information taken from other websites, but they allow users to get the answers they seek without ever clicking away from Google. Some users are now bypassing search engines altogether, posing questions directly to tools like ChatGPT.<br>The erosion of website traffic is a problem for everyone, explains Soheil Ghili of Yale SOM. If there is no financial incentive to produce new content, large language models will have only outdated information to draw on for training and/or grounding, and the models’ usefulness will suffer. “A product that doesn’t have a market doesn’t get produced, at least not in an efficient amount,” Ghili says. As a result, “we might just get both lower-quality and lower-quantity content.”<br>One solution to this problem is a bulk deal. Reddit, for example, has licensed its content to both Google, charging a reported $60-70 million per year for the privilege of crawling its content. But that playbook won’t work in most situations, Ghili points out: “You cannot go around and negotiate a deal with every single small website.”<br>Pay-per-crawl is a new approach, pioneered by companies including Cloudflare and Tollbit, that allows content producers to charge AI crawlers an access fee for visiting a page. Figuring out how to determine the right fee is the subject of a new working paper by Ghili, Nima Haghpanah of Yale SOM, and graduate student Richard Archer.
“Pay-Per-Crawl Pricing for AI: The LM-Tree Agent”
Publishers could, of course, charge AI crawlers the same price for every piece of content on their website, but that’s probably not the best strategy—it’s reasonable to assume content varies in its usefulness for large language models. But it’s also not feasible for publishers to manually price every piece of content they produce.<br>Using real content from a German technology publisher called HardwareLuxx, Ghili and his collaborators developed an AI tool that can, at scale, help publishers find the optimal pay-per-crawl price for each article—one that will maximize revenue while remaining palatable to the crawlers. This agent, which they call the LM Tree, identifies the attributes that make content valuable and sets prices based on those characteristics.<br>To put the LM Tree to the test, the researchers built a simulated market on top of HardwareLuxx’s real articles. Each article was assigned a hidden value—what an AI crawler would be willing to pay to access it—and the agent never saw that value directly. All it received was binary feedback: at a proposed price, would a crawler buy this article or not? From nothing but the article’s text and a handful of these accept/reject signals, the LM Tree had to work out what each piece of content was worth.<br>The researchers compared the LM Tree against strategies that price content using only the website’s structured information—the topic categories HardwareLuxx already assigns, like “graphics cards” or “processors.” The LM Tree’s advantage is that it also reads the prose, and from the text it discovers pricing rules that no column in the dataset contains.<br>One example: HardwareLuxx’s taxonomy has a category for graphics cards, but nothing that singles out flagship graphics cards. The agent learned on its own that articles about high-end, flagship GPUs should command higher prices—even though “this is about a flagship GPU” is not a field anywhere in HardwareLuxx’s data. Strikingly, the word “flagship” never has to be defined in advance for the rule to work. A large language model’s general world knowledge lets it recognize a flagship GPU when it reads one, which means the same rule can automatically tag any future article—including content the agent has never seen—without anyone writing down a list of what counts as “flagship.”
Looking for more insights?<br>Sign up to get our top stories by email.
Thanks for signing up
Next, the researchers set the LM Tree loose on a new set of articles and compared its revenue to other pricing strategies. Charging the same price for every article performed worst. Pricing by format (news or review) did better. The LM Tree’s custom, text-based prices did...