AI companies must buy in
For Cloudflare’s plan to work, AI companies must sign up, too. However, while some AI companies may not see the incentive, Cloudflare has confirmed that it has partnered with AI companies on the initiative, which may benefit from having a simple interface to negotiate with content creators.
Cloudflare suggested its AI partners benefit from “long-term collaboration” with creators whose updated content will help AI products stay relevant. They also can stop wasting money scraping poor quality data sources, a Cloudflare blog said.
“Without ongoing contributions from content creators, AI systems risk becoming outdated, biased, or less reliable—ultimately diminishing user trust and the value of AI products,” the blog said. “Cloudflare is working with AI companies to give them more signals, and ultimately improve the quality and relevance of content they can access. A healthy, sustainable ecosystem of original content is critical for AI innovation and relevance.”
However, Cloudflare’s gamble seems to depend on AI companies agreeing to pay the prices set by publishers, and that could potentially scramble the experiment if bidding wars reduce rates to the point that they alienate publishers. It also hinges on Cloudflare detecting the AI bots, which, for now, relies on user reports and Cloudflare’s analysis of mass traffic patterns.
“In the early days, price discovery will play a key role—as creators gain data on whoʼs paying for what, a transparent market will emerge that reflects the true value of original content,” Cloudflare said.
Looking to the future, Cloudflare suggested that its pay-per-crawl system would “evolve significantly.” Perhaps one day publishers could use it to “charge different rates for different paths or content types,” potentially even introducing dynamic pricing in the AI scraping environment. In that future, Cloudflare predicted that AI companies would possibly be incentivized to create agents that would crawl the web, seeking the best content deals to support specific AI products.