The global collection of hundreds of millions of publicly accessible websites crammed with information, which many of us take for granted, is undergoing its most significant change in 20 years.
Enormous and robust, the “open web” ecosystem is also vulnerable. Its sites generally rely on a single search engine, Google, for users to find them.
The system works on the idea of reciprocity: Websites publish content optimised to ensure a top ranking on Google, which sends traffic, which generates advertising revenue, which funds more content. And so on.
Now, this basic assumption has been upturned.
Over the past year, Google has rolled out AI Overviews around the world, so that when users type a query into Google, they often receive an AI-generated answer above the standard blue links.
Today, the search giant went one step further, launching in Australia the chatbot-like search experience “AI Mode”, with no blue links and fewer clickable results.
Google is morphing from “search engine” to “answer engine”, from an index of third-party information to a publisher of AI-generated summaries.
And the sites on the frontline of this change are news sites, like ABC News.
News sites in the US and the UK have reported a sudden and sustained decline in traffic to articles from readers arriving via search engines, to the extent that some publishers are laying off workers and warning of an “existential crisis” for journalism.
But in Australia, the “AI Armageddon for online news publishers” hasn’t generated too much fuss.
What’s going on?
One answer is the news sites are doing fine. The industry standard for online audience measurement, Ipsos Iris, shows some top-10 news sites have swapped rankings, but their combined readership has actually gone up in the past year.
But another measure of online readership tells a different story.
Exclusive data obtained by the ABC from SimilarWeb, a rival service to Ipsos, shows the combined readership of the top news sites in Australia is much lower than a year ago.
And for smaller publishers reliant on Google for a greater share of their traffic, the declines may be even greater.
The challenge of measuring online readership
SimilarWeb data is widely used outside of Australia to track the rankings of news sites and detect trends in readership. Its data has informed reporting around the impact of Google AIO, from Bloomberg to the Wall Street Journal.
Ipsos and SimilarWeb gather their data in different ways, with neither having a perfect view of who’s visiting what sites and how they’re getting there.
Is the simple Google search now less useful?
Ipsos said it measures activity on thousands of devices and then extrapolates the results across a population to provide an “independent source of truth for the media industry”.
SimilarWeb aggregates data from many sources, including internet service providers, browser plug-ins and mobile apps.
“You’ve effectively got multiple different ways of looking at the data but not all of them are showing you the complete picture in my opinion,” Laurence O’Toole, CEO of the UK SEO company Authoritas, said.
“I think SimilarWeb data is probably the most robust for a view across the industry.”
As well as showing declines in monthly audience sizes for the top news sites, the SimilarWeb data suggests changes to Google Search (which accounts for over 90 per cent of the search engine market in Australia) may be a cause.
Fewer readers are being directed to articles after clicking on a search result, the data shows.
Most Australian news sites are seeing declines in search traffic, with one down by 35 per cent.
News sites in US, UK reporting declines in traffic
The measured declines in audience size and search traffic match what publishers are independently reporting overseas, Mr O’Toole said.
“The [SimiliarWeb data] is consistent with news sites that we’re working with and what I am hearing across the industry.”
Lily Ray, an American SEO expert, said: “Most of the news sites I have spoken to or worked with … have seen a pretty substantial decline in organic search traffic from Google in the past year.”
“The dramatic drop-off has been shocking and disorienting for many site owners.”
In the year before AI Overviews, combined search traffic to news sites was fairly steady, although a series of Google algorithm updates had seen smaller publishers in particular lose referrals.
The SimilarWeb data also fits the emerging understanding of the prevalence of AI summaries and how they affect search behaviour.
Over a third of Google searches in Australia now result in an AI-generated summary and Google users who encounter an AI summary are much less likely to click on a standard result link.
Joanne Kuai, an RMIT researcher with a focus on AI, search engines, and journalism, said for these reasons she would “expect to see declines” for news sites.
“Generative search engines reduce the incentive to click through. That inevitably cuts traffic to publishers.”
Why are some sites more affected than others?
One reason some sites are more affected may be the kinds of articles they publish, said Kevin Indig, a US SEO expert who’s looked closely at global SimilarWeb data on the impact of AI Overviews.
Evergreen content such as how-to guides, recipes or answers to common recurring questions are what “suffer the most under Google’s AI answers”, he said.
Loading
Features or coverage of breaking news (sometimes called “hard news”) may be less easily summarised.
“Looking deeper into the data, the conclusion I’ve come to is that the news side of the house is fine,” he said.
“Where everybody’s seeing a drop is with evergreen content.”
Our AI future isn’t conscious robots with lethal ambition
The company with the best idea of what’s happening is Google itself, which collects extensive data on how users interact with AI-generated summaries.
But so far Google has not shared this knowledge with publishers or the broader public.
Google declined the ABC’s invitation to comment, with a spokesperson directing the ABC to an August blog post that stated “total organic click volume from Google Search to websites has been relatively stable” but did not provide data.
In general, Google says AIO makes the search experience better, which is a win-win for itself and publishers. More queries means more clicks to websites, and these clicks are “higher quality”, meaning users are staying on the websites for longer.
But publishers in Australia and elsewhere say that hasn’t been their experience.
Small, independent news sites may be worst affected
The men’s lifestyle site Man of Many, which publishes product reviews, interviews, features and advice, saw a dip in search traffic following the rollout of Google AIO in Australia, director Scott Purcell said.
Traffic to “informational” articles was down 10–30 per cent.
Other publishers were seeing similar, he said.
“Privately, I’ve had some publishers tell me they’ve had to lay off a lot of staff just because they’re not generating as much traffic as they previously did.”
Publishers were keeping quiet to avoid scaring advertisers, he said.
“It doesn’t make sense from a commercial perspective for a media publisher to announce to its advertisers that it’s getting lower traffic.”
Dancing to Google’s tune is nothing new for publishers. The tech giant regularly makes aggressive changes to the algorithm that determines search rankings, and which ultimately decides which websites get readers and revenue.
But AIO is different for three reasons. First, it’s having a bigger impact. Second, that impact is felt more widely, by more websites. And third, Google’s AI has been trained on the content of the same news sites that are now losing traffic.
Because of this, publishers are taking legal action.
In July, the Independent Publishers Alliance (IPA) in the UK filed a complaint with competition regulators of the UK and the European Union accusing Google of using publishers’ content (in AI-generated summaries) at a cost to these sites.
This month, it said one-third of small publishers in the UK could be out of business by the end of the year.
“Be prepared to hear Google’s narrative of ‘AI search drives better quality traffic,'” IPA managing Chris Dicker told the ABC, as a warning to Australian publishers.
“That certainly could not be further from the truth, the metrics that we measure for business success currently suggests the complete opposite.
“Start planning for a world without any Google traffic.”
The Australian equivalent of the Independent Publishers Alliance, the Digital Publishers Alliance (DPA), says it’s keeping open the option of legal action.
“We reserve all our rights to take any [legal] actions here in Australia,” DPA chair Tim Duggan said.
“Instead of bringing customers back to a publisher’s site for them to monetise it themselves, [AI] steals the content and delivers it to a user without any compensation or reward for the original content creator.
“This theft of content from the people who created it is not sustainable, and is a slippery slope that removes many of the incentives to create original content.
“We firmly believe that publishers who create news and information that’s scraped by AI companies should be fairly compensated for the use of their work.”
Google launches chatbot-like ‘AI Mode’ in Australia
Having spent decades developing the open web ecosystem that rewards news sites with readers, Google appears determined to switch course, rolling out AI tools that keep many users milling about in Google’s walled garden.
It launched its “AI Mode”, a chatbot-like search tool in Australia today after being available in most other countries for several months.
Users can now tap an “AI Mode” button to the right of the Google search bar, or on the home screen of the Google app, or go to google.com/ai.
The new “AI Mode” button to the right of the Google search bar. (Supplied: Google)
Users interact conversationally with the search bar through text, voice or taking a photo or uploading an image, and are able to ask long-form follow-up questions, similar to ChatGPT. It can recall past conversations without prompting.
There are usually no standard blue links, so that it refers less traffic to news or other websites than even AI Overviews (AI Mode doesn’t pass referral data, so publishers don’t actually know how many clicks they’re getting).
If there’s not high enough confidence in the quality or helpfulness of an AI response, AI Mode will provide a set of links.
The “total reimagining of web search” has been described as more like a “zero-click branding channel” than a traditional search engine.
Google warns that “as with any early-stage AI product, AI Mode doesn’t always get it right.”
Loading
If it becomes the default search option (as predicted), traffic to news sites could fall further again. The open web could wither. Today’s abundance of news sites may come to seem a quaint anachronism, like MySpace or GeoCities.
“We are witnessing an acceleration of platform monopolisation and concentration of power, now fuelled by AI technologies, and this is deepening the crisis of journalism,” RMIT’s Dr Kuai said.
“If press publishers are driven out of the market by worsening platform monopolies, we risk losing diverse perspectives, marginalised voices, and investigative reporting.
“That weakens the information system, making societies more vulnerable to authoritarian tendencies.”
But questions remain. Google makes most of its money selling ads on the type of websites it’s no longer directing the same proportion of readers towards.
It’s not clear how Google can balance giving users what they want (quick summaries as answers) and driving clicks to Google Ads, SEO expert Lily Ray said.
“Personally, I find it hard to imagine that Google can be as profitable as it is now.”
Kevin Indig sees at least two possible outcomes.
News sites could block Google, having decided they’re not getting enough readers in exchange and Google was effectively behaving as a rival stealing their content.
Or Google could ditch the news websites.
At some point, Mr Indig said, “the AI models may be so well trained they don’t need the content anymore.”
“Maybe it’s enough to have a few licensing partnerships with a few select publishers.”
“And you don’t need anything else.”