We are excited to announce that Project Galileo will now include access to Cloudflare’s Bot Management and AI Crawl Control services. Participants in the program, which include roughly 750 journalists, independent news organizations, and other non-profits supporting news-gathering around the world, will now have the ability to protect their websites from AI crawlers—for free.
Project Galileo is Cloudflare’s free program to help protect important civic voices online. Launched in 2014, it now includes more than 3,000 organizations in 125 countries, and it has served as the foundation for other free Cloudflare programs that help protect democratic elections, public schools, public health clinics, and other critical infrastructure.
Although we think all Project Galileo participants will benefit from these additional free services, we believe they are essential for news organizations.
News organizations, particularly local news, are facing significant challenges in transitioning to the AI-driven web. As people increasingly turn to AI models for information, less of their web traffic is making it to the actual website where that information originated. Industries, like news organizations, that rely on user traffic to generate revenue are increasingly at-risk.
Allowing news organizations to monitor and control how AI crawlers are interacting with their websites, will help them better protect their content and make more informed decisions about engaging with AI companies. Ultimately, our goal is to provide the tools news organizations need to negotiate fair compensation for their work.
AI is fundamentally changing how traffic flows on the Internet. Cloudflare recently published data that shows with Open AI its 750 times more difficult for website owners to get the same volume of traffic than it was with previous Google search. With Anthropic, it’s 30,000 times more difficult.
News organizations rely on traffic to not only connect with their readers, but also generate revenue from subscriptions, advertising, e-commerce, and licensing. The CEO of the Financial Times recently stated that AI had caused a ”pretty sudden and sustained’ decline of 25% to 30% in traffic to its articles arriving via search engines.”
Potential losses of user traffic and revenue come at an already precarious time for the news industry. It is well-documented that small, independent newspapers and news radio stations continue to face significant financial pressure, particularly in the United States. According to recent US Congressional testimony, more than two newspapers closed per week in 2024 with one third of the country’s newspapers set to close before the beginning of 2025. A 2024 report by the Northwestern Local News Initiative reported more than 206 US counties were without any local news source, and 1,561 had only one.
Recent funding cuts to the Public Broadcasting Corporation and National Public Radio, which provided grants, programing, and other support to public news stations around the US, have put further strain on these organizations with more closures expected.
Giving control back to journalists
An important first step in helping journalists and news organizations adapt to the AI-driven web is providing tools to help them monitor and control AI models’ access to their content.
“In an era defined by AI and digital disruption, providing robust tools to independent media isn’t just support – it’s a lifeline” – Meera, CEO Internews Europe
“Independent publishers need tools that are easy to use and affordable, so they can focus on growing their business. LION appreciates the security and protection Cloudflare has provided our members through Project Galileo for years, and we’re excited to see more resources now available to help members manage the rapidly evolving landscape of digital security.” – Sarah Gustavus Lim, LION Membership Director
Cloudflare Bot Management and AI Crawl Control were designed for exactly these purposes. Bot management is a security tool that uses machine learning to analyze web traffic to distinguish between good bots, like search engine crawlers, and bad bots that attack websites or steal credentials. It allows website owners to block bad bots from reaching their websites, while making sure helpful bots can continue to do their work.
AI Crawl Control provides similar tools to identify and manage AI crawlers. Cloudflare uses a variety of techniques to identify and categorize crawlers (HTTP header, heuristics, and other behavior) giving website owners the ability to analyze their activity by type (e.g. AI search, AI scraper), where they are coming from (Google, OpenAI, Anthorpic, etc.), and what content they are accessing. Here’s the kind of data that Cloudflare’s AI Crawl Control tool can provide (using the radar.cloudflare.com domain) as an example:
Cloudflare combines these insights with easy-to-use controls that allow website owners to make informed decisions about whether to make their data available, including to only certain types of bots or to individual AI companies. This would, for example, allow a local newspaper to decide to block all AI crawlers and maintain direct connection to their readers via their own website, block only AI scrapers while allowing AI search crawlers that refer traffic, or negotiate and sell exclusive access to their content to a single AI company. The following image shows how AI Crawl Control lets users allow or block access on a crawler-by-crawler basis:
We think the ability to control and monitor AI crawler activity will provide immediate help to news organizations looking to protect their content and understand how models are using their data.
“Independent publishers need tools that are easy to use and affordable, so they can focus on growing their business. LION appreciates the security and protection Cloudflare has provided our members through Project Galileo for years, and we’re excited to see more resources now available to help members manage the rapidly evolving landscape of digital security.” – Sarah Gustavus Lim, LION Membership Director
We also think it will provide longer term insights that will allow news organizations to negotiate mutually beneficial relationships with AI companies over time.
“Independent media’s ability to fulfill its democratic function by gathering news and distributing trusted information depends on generating revenues free from political or business influence. By monitoring and monetizing the crawling of publisher’s sites, media can protect their intellectual property while developing new revenue streams to support their quality journalism.” – Ryan Powell, Head of Innovation and Media Business at International Press Institute
A free press, if we can keep it
Journalism is part of the foundation of free society and democratic governance. It helps hold power accountable and provides a voice to the marginalized and underrepresented. It also protects the free and open markets that allow startups to challenge powerful incumbents.
Local news in particular helps create shared identity. Not only by covering community events, high school sports, farmers markets, and new businesses, but also providing essential transparency and oversight over local officials, school boards, public safety events, and elections.
Helping protect journalists and news organizations online has always been part of Cloudflare’s mission. We see it as essential to our business and the future of the Internet.
If you are interested in learning more about Project Galileo, sign up today. If you are interested in helping build a better Internet, come join us.