“For decades, the Internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. This cycle rewards creators that produce quality content with money and a following, while helping users discover new and interesting information,” Cloudflare said in its announcement. “That model is now broken. AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source — depriving content creators of revenue, and the satisfaction of knowing someone is reading their content. If the incentive to create original, quality content disappears, society ends up losing, and the future of the Internet is at risk.”
Pay per crawl
Cloudflare is testing a new mechanism payment mechanism, pay per crawl, that enables website owners to decide whether they will permit AI crawlers to access their content, and if that access will be free or they will charge for it. The technology, now in private beta, integrates with existing web infrastructure to create a framework to enable site owners to require payment, and tell the crawler the price via an HTTP “402 payment required“ response code.
The site owner can currently set a single price for the site or choose to let certain crawlers access it at no charge, but Cloudflare expects the feature to evolve over time, perhaps to allow dynamic pricing, or charge different amounts for various types of content.