Crawl waste – basically, cluttering your crawlable content with tons of non-indexed pages – is one of the most insidious SEO pitfalls.
We know (because we do it all the time for my agency’s clients) that reducing the number of non-indexed pages and highlighting richer, authoritative content is a factor in improving rankings and visibility in traditional SERP listings.
But how does it impact AI search visibility? And were the results different by platform?
Thanks to some recent cleanup and data-crunching (we used Profound to measure changes in AI search visibility) for a client in the childcare vertical, we’ve got some directional answers and takeaways that I’ll lay out in this article.
Yes, it’s just one website, but with AI moving at light speed, having promising results that need further validation is a valuable first step.
Crawl waste cleanup: What we did
Sitemaps should only feature canonical versions of pages that a brand wants indexed in search results.
The childcare client’s site had 2,000 indexed pages and ~45,000 pages that were not being indexed – including 4,000 duplicate and redirected URLs incorrectly listed in the sitemap. (A sitemap should only contain canonical URLs you want Google to find and index.)
The site also contained low-quality directory listings, all of which pointed to poor health.
We optimized the sitemap and indexing strategy by reducing the number of duplicate and redirected URLs in the sitemap from more than 4,000 to fewer than 200.
This cleanup improved crawl efficiency, helping search engines and AI systems focus on the client’s high-quality, intent-aligned content.
Dig deeper. Crawl efficacy: How to level up crawl optimization
Get the newsletter search marketers rely on.
How the AI search platforms responded
The topline:
This initiative lifted visibility across every AI search platform – up to 10% in some cases – pushing overall visibility above 90% for Perplexity, AI Mode, AI Overviews, Gemini, Grok, and Meta AI. (We calculate visibility by dividing the number of times the brand appeared in prompt results by the total number of relevant prompts.)
The one exception was ChatGPT, which saw a 4.1% drop. My take is that ChatGPT can be slower to absorb the latest online information unless users enable web access and browsing. Because the cleanup involved major indexing and technical changes, ChatGPT may have lagged compared to faster adapters like Perplexity and Google.
The entire Google suite of AI tools responded strongly to the cleanup.
That result – while just one example – aligns with Google’s repeated message that what benefits traditional SEO also benefits AI search.
Even with ChatGPT’s decline, overall visibility still grew by 3.5%.
It isn’t a massive jump, but it signals an upward trajectory that helps the brand gain ground on competitors.
In Gemini, the lift was enough for the client to finally overtake a rival that had been consistently ahead.
Dig deeper: Crawling for AI search: Balancing access, control, and visibility
Takeaways – and planning more quick wins
This case study isn’t the final word, but it points to clear next steps and quick wins worth pursuing.
Be diligent about gauging the impact of AI search initiatives, even those rooted in more traditional SEO strategies.
Anything that makes it easy for search platforms – AI and otherwise – to discover your best content is worth prioritizing.
A clear URL structure and straightforward, intentional information architecture (IA) should help you with discovery across platforms.
If you need an IA model, build a major pillar about a topic, write a wealth of authoritative content on that topic, and link to it from the website and external sources favored by LLMs, including Reddit.
Never assume that all AI search platforms behave the same.
Look for major variances between (say) ChatGPT and the Google AI suite. Be ready to run a 2.0 analysis to determine which platform(s) should take precedence.
Always aim for validation beyond a single test or case study. Consider initial findings an important first step in understanding the greater significance of AI search optimization techniques.
Contributing authors are invited to create content for Search Engine Land and are chosen for their expertise and contribution to the search community. Our contributors work under the oversight of the editorial staff and contributions are checked for quality and relevance to our readers. Search Engine Land is owned by Semrush. Contributor was not asked to make any direct or indirect mentions of Semrush. The opinions they express are their own.