Google sends publishers 831x more visitors than AI systems, according to TollBit’s State of the Bots Q2 2025 report. Even though AI referrals barely register next to Google’s, bots are hammering publisher sites — pushing costs higher, scraping more content, and often ignoring rules meant to limit them.
Scraping up, referrals down. AI is scraping the web, but it still isn’t sending meaningful visitors back.
Google referrals dropped from 90%+ of all external traffic in Q2 2024 to 84.1% in Q2 2025.
AI apps sent just 0.102% of referrals.
For every visitor from an AI system, Google delivered 831.
Click-through rates from AI interfaces were weak – 91% lower than top-10 organic search CTRs. It took about 135 AI scrapes to produce a single human referral.
Publishers with OpenAI licensing deals saw 88% more scraping and significantly stronger referral rates than those without.
More bots up, less humans. AI bots are becoming a dominant force in web traffic, even as they deliver no value to publishers.
Human visitors to TollBit-tracked sites fell 9.4% from Q1 to Q2 2025, while AI traffic rose.
At the start of 2025, 1 in 200 visitors was an AI bot. By Q2, it was 1 in 50 – a 4x increase.
AI bot traffic has now surpassed Bingbot, the world’s second-largest search crawler.
Google’s AI Overviews expansion last October drove a 34.8% increase in Googlebot crawls, but the crawl-to-referral ratio worsened by 24.4%.
Content demand shifts. AI tools are shifting consumer behavior, answering questions directly instead of sending users to publishers.
B2B/professional content saw the highest scraping levels relative to human traffic.
The fastest AI request growth was for parenting (333%) and deals/shopping (111%) content.
National news got 5x more real-time RAG scrapes than training crawlers.
APAC sites were hit the hardest, receiving 3x more requests than U.S. sites, while European sites saw 27% fewer AI requests.
Bot wars escalate. Publishers are pushing back, but many AI bots are still breaking (or rewriting) the rules.
Publisher blocking AI bots surged 336% YoY.
TollBit’s Bot Paywall hits grew 360% between Q1 and Q2 2025.
13.26% of AI bot requests ignored robots.txt in Q2 2025, up from 3.3% versus Q4 2024.
OpenAI’s 404 error rate jumped from 0.3% to 3.7%, often due to hallucinated URLs.
Anthropic’s Claude error rate fell sharply after gaining live web access (from 55% in Q2 2024 to 4.8%).
Why we care. The open web appears to be in trouble. Human visitors are declining, while AI bots are multiplying, scraping more content, and sending back almost no traffic (which is why we’ve seen Cloudflare’s pay-per-crawl and RSL initiatives). If this continues, publishers will likely face higher costs and fewer returns (e.g., less clicks, customers, and revenue), which threatens the web’s business model.
Methodology. TollBit analyzed web traffic across its partner publisher network, tracking billions of requests to measure human visitors, AI bot activity, referral rates, and server responses from Q1–Q2 2025.
The report. State of the Bots Q2 2025 (registration required)
Search Engine Land is owned by Semrush. We remain committed to providing high-quality coverage of marketing topics. Unless otherwise noted, this page’s content was written by either an employee or a paid contractor of Semrush Inc.
