OpenAI Crawl Activity Triples Post-GPT-5 While AI Overviews Cut Organic Clicks 38% | SEO Data Briefing

1. OpenAI's Crawler Surge: 7 Billion Events Tell the Story

Botify analyzed roughly 7 billion OpenAI bot log events from November 2024 through March 2026 across their enterprise client base. The finding is blunt: OpenAI's search-oriented crawler exploded after GPT-5 launched in August 2025.

The more consequential finding is the ratio change. Before GPT-5, search events to training events ran at 0.95:1, a wash. After GPT-5, that flipped to 1.14:1. OAI-SearchBot now outpaces GPTBot in log volume. OpenAI is crawling more for live search retrieval than for model training. That's a structural change in how they use the web.

ChatGPT-User, the bot that fetches pages when users paste URLs directly into ChatGPT, dropped 28% between December 2025 and March 2026. As SearchBot scales retrieval-augmented generation, user-triggered fetches become redundant.

Industry-Level Crawl Breakdown

The vertical-level data is where practitioners should pay attention. OAI-SearchBot is not hitting every industry at the same rate:

OpenAI vs. Googlebot: Scale Comparison

The growth is real, but perspective matters. In the most recent 30-day window:

OpenAI sits at roughly 4% of Google's crawl volume and 14% of Bing's. That 4% is nearly triple last year's 1.38%. Keep the growth rate steady and OpenAI's crawl footprint intersects Bingbot's within 18 to 24 months. At that point, large publishers need a real answer on crawl budget allocation, not just a Googlebot strategy.

2. AI Overviews Cut Organic Clicks 38%: The Field Study

The second major data release this week comes from a randomized field study that offers the most methodologically sound measurement of AI Overviews' traffic impact to date. Earlier observational work leaned on rank trackers or aggregate analytics. This one used a controlled experimental design.

Study Design

The study ran January through February 2026 with 1,065 U.S. Chrome desktop users split into three groups: a control group using normal Google, a "Hide AIO" group with AI Overviews suppressed via browser extension, and an AI Mode group. Key methodological details:

• Over 95% of users in the Hide AIO group never noticed AI Overviews were gone, confirming the intervention didn't change user behavior artificially
• Pre-registered with the AEA RCT Registry, though not yet peer-reviewed
• Two-week observation window per participant

The Numbers That Matter

Remove AI Overviews and outbound clicks per search jump from 0.38 to 0.61, a 60% increase in clicking. The effect concentrates at the top position: top-position AI Overviews appeared in 85% of AIO instances and nearly doubled outbound clicks when removed. Lower-position AI Overviews showed no measurable suppression at all.

Sponsored click volume held steady across all groups. Search frequency was unchanged. AI Overviews are cannibalizing organic clicks , not paid clicks and not overall search behavior.

3. Google's $60.4 Billion Revenue Paradox

That click-loss data landed the same week Alphabet reported Q1 2026 earnings. Google Search revenue hit $60.4 billion, up 19% year-over-year. Total Alphabet revenue reached $109.9 billion, up 22%.

CEO Sundar Pichai tied the growth directly to AI: "People love our AI experiences like AI Mode and AI Overviews, and they're coming back to Search more."

Other operational figures from the earnings call:

• Search latency cut by more than 35% over five years, including through AI feature additions
• AI response costs down over 30% since upgrading to Gemini 3
• Strong verticals: retail, finance, and health
• "Queries are at an all-time high" (Pichai)

Google disclosed none of the following on the earnings call: AI Overviews click-through rates, AI Mode revenue attribution, or whether publishers see net traffic gains or losses from AI-influenced results. The gap between Google's revenue line and publisher traffic is the defining tension of 2026 search.

4. How Five AI Search Engines Cite Differently

A comparative citation analysis across ChatGPT, Google AI Overviews, Google AI Mode, Gemini, and Perplexity found that these engines draw from very different source pools, yet converge on one behavior: citing established brands.

Source Overlap Between Engines

Citation source agreement between any two engines runs from just 16% to 59%. The highest overlap is between Google AI Mode and AI Overviews at 59%, which makes sense given shared infrastructure. Brand citation overlap is tighter, from 36% to 55%, confirming that brand authority carries across AI search contexts regardless of engine.

A few patterns stand out:

• Gemini and Perplexity pull heavily from institutional and authoritative sources; AI Overviews trusts user-generated content at a meaningfully higher rate
• .edu domains underperform across every engine, hitting 0 to 3.2% citation share
• AI Overviews sources 10.6% of citations from a single video platform and 2.9% from a single forum platform
• ChatGPT shows the widest source variety, making it hardest to tune for and potentially the most meritocratic of the five

5. Bing Previews Citation Share: The First Competitive AI Visibility Metric

At SEO Week in New York City on April 27, 2026, Microsoft's Krishna Madhavan previewed four new AI reporting features coming to Bing Webmaster Tools:

1. Citation Share — the percentage of citations your site captures within a specific grounding query, giving competitive context against other cited sources
2. Grounding Query Intent Labels — classifies queries into 15 predefined intent categories including Learning, Informational Search, Navigational, Research, Comparison, Planning, Conversational, Content Filtered, and more
3. Grounding Query Topic Labels — groups queries under topic classifications
4. GEO-Focused Recommendations — guidance covering content structure, crawlability, indexing and canonicalization signals, structured data adoption, and structured data quality

Citation Share is the one that changes measurement. Current AI search analytics tell you how many citations you got, not what share of available citation slots you captured. Citation Share introduces competitive benchmarking: are you the dominant source for a query, or one of ten?

6. Cloudflare's Agent Readiness Score: Most Sites Aren't Prepared

Cloudflare Radar's Agent Readiness score, announced April 17, scores how well websites support AI agents across four dimensions. Their analysis of the 200,000 most visited domains globally is not encouraging:

The four scoring dimensions:

• Discoverability: robots.txt, sitemaps, Link headers
• Content Accessibility: Markdown support (text/markdown content negotiation)
• Bot Access Control: Content Signals, Web Bot Auth
• Capabilities: APIs, MCP servers, OAuth discovery, Agent Skills

Cloudflare's own optimized documentation produced 31% fewer tokens consumed and 66% faster response times when accessed by AI agents compared to non-optimized technical sites. Agent readiness is not just an access control question. It directly affects the quality and cost of AI-generated responses that cite your content.

Two related Cloudflare releases are worth tracking. Their Redirects for AI Training feature (April 17) lets operators redirect verified crawlers to canonical pages with a single toggle, so AI systems ingest current content rather than stale versions. Their Moving Past Bots vs. Humans post (April 21) argues that anonymous credentials, not binary bot/human classification, are where web access control is heading.

7. Strategic Implications: What to Do With This Data

For Publishers and Content Sites

• Audit your crawl logs now. The Botify data shows OpenAI's crawl budget shifted from training to search retrieval. If you're in healthcare or media, you may be feeding ChatGPT search results without knowing it. Check for both OAI-SearchBot and GPTBot in your server logs.
• The 38% organic click decline is structural, not cyclical. AI Overviews appear on 42% of queries and that share will grow. Plan for top-position organic results receiving 38% fewer clicks on AIO-triggered queries. Diversify traffic sources now, not after the next algorithm update.
• Invest in brand signals. The citation pattern data shows 36-55% brand overlap across all five major AI engines. Brand authority is the one optimization that transfers everywhere.

For Technical SEOs

• Agent readiness is the next technical SEO frontier. Only 4% of top sites declare AI usage preferences and 3.9% support Markdown. Early movers have a clear gap to exploit. Implement text/markdown content negotiation, add Content Signals to robots.txt, and evaluate MCP server support.
• Watch the Bing Citation Share rollout. Once available, this is the first quantitative metric for competitive AI visibility. Get your Bing Webmaster Tools instrumented before it ships.

For Executives and Strategists

• Google's revenue growth and publisher traffic loss are two sides of one coin. Longer AI-enhanced sessions mean more ad impressions and more revenue for Google, and fewer clicks out to publishers. The $60.4B quarter proves the model works for Google. The field study proves it costs everyone else.
• OpenAI's crawl footprint is no longer a rounding error. At 4% of Googlebot volume and growing 3x year-over-year, it belongs in your crawl budget planning today.

Frequently Asked Questions

About the Author

About the author

Francisco Leon de Vivero

Francisco is a senior SEO strategist and VP of Growth at Growing Search, with 15+ years of enterprise search experience. He previously served as Head of Global SEO Framework at Shopify from 2015 to 2022 and focuses on technical SEO, international search strategy, and platform optimization.

SEO Francisco LinkedIn YouTube