Common Crawl AI Visibility Audit: Check If LLM Crawlers Can See Your Site

There is a good SEO tip going around: use Common Crawl to check whether your site is in the open-web data layer that many AI systems may build from.

The tip is worth stealing. The claim needs tightening.

Common Crawl's AI Visibility Audit pushes SEOs to check one layer upstream from rankings, prompt tracking, and AI citations: crawler access. If the crawl cannot reach your pages, those pages have a weaker shot at appearing in future training datasets, retrieval systems, citation graphs, and AI answer surfaces.

So yes, run the check. Just do not oversell what it proves.

The Five-Minute Common Crawl Check

Open your terminal and replace example.com with your domain.

DOMAIN="example.com"

# 1. Check the live robots.txt
curl -s "https://$DOMAIN/robots.txt"

# 2. Test whether a CCBot-style request gets blocked at the edge
curl -A "CCBot/2.0" -I -L "https://$DOMAIN/"

# 3. Find the newest available Common Crawl index
curl -s "https://index.commoncrawl.org/collinfo.json" | python3 -c \
'import json,sys; print(json.load(sys.stdin)[0]["id"])'

# 4. Query the current index for the domain
INDEX="CC-MAIN-2026-25"
curl "https://index.commoncrawl.org/${INDEX}-index?url=${DOMAIN}/*&output=json" | head

If rows come back, Common Crawl has captures for that domain in that snapshot. If nothing comes back, you have a crawl availability question to investigate.

What This Test Actually Proves

The Common Crawl index test answers a specific question:

Did Common Crawl record URLs from this domain in this crawl snapshot?

That is not the same as:

• "A specific LLM trained on this page."
• "This page will appear in AI answers."
• "This domain has strong AI visibility."
• "The content was fully understood, parsed, or cited."

Think of it as a crawl-layer diagnostic. It tells you whether the page made it into a public corpus that can feed research, training, evaluation, analysis, and retrieval workflows. That matters. It is also only one piece of the AI visibility stack.

I would put it beside log files, server-rendered HTML checks, structured data review, internal link analysis, and AI citation tracking. It pairs especially well with the crawler logic in AI crawler ghost citations and the agent-readable content checks in llms.txt and Lighthouse Agentic Browsing.

The SEOFrancisco Result

I ran the check against SEOFrancisco on July 10, 2026.

That is a clean pass for the basic access test. It does not mean every page is represented, that every capture is useful, or that a model trained on those URLs. It means the domain is visible in the public Common Crawl index and is not currently blocking a CCBot-style request.

How To Read The JSON Response

A Common Crawl index row usually includes fields like these:

{
  "urlkey": "com,seofrancisco)/insights/example",
  "timestamp": "20260617165032",
  "url": "https://seofrancisco.com/insights/example/",
  "mime": "text/html",
  "status": "200",
  "digest": "...",
  "length": "15531",
  "filename": "crawl-data/CC-MAIN-2026-25/segments/..."
}

The useful fields for a quick SEO read are:

• url: the captured URL. Check whether it is the canonical host and path you care about.
• timestamp: when Common Crawl captured it. This helps separate old crawl presence from current crawl presence.
• status: the HTTP status Common Crawl recorded. A 200 is cleaner than a redirect, soft block, or error.
• mime: the content type. Most editorial pages should show as HTML.
• length: a rough size signal. Very small HTML can hint at thin captures, redirects, shells, or blocked content.
• filename: the WARC file where the capture lives, useful if you later want to retrieve the archived record.

The Common Crawl index documentation explains that the index points to records inside WARC files. For this article's quick audit, you do not need to pull the WARC. You just need to know whether the crawl found real pages and what status those pages returned.

Why The CCBot Check Matters

The index query tells you what Common Crawl has already captured. The CCBot request tells you what your site allows right now.

Those can disagree.

A site might be present in CC-MAIN-2026-21 because it was crawlable in May, then missing later because a CDN setting, firewall rule, or managed robots.txt change blocked AI crawlers. The reverse can happen too: a site might be open today but absent from a recent snapshot because it had not been discovered, prioritized, or recrawled yet.

Common Crawl's CCBot documentation also warns that some crawlers falsely identify as CCBot. If you are reviewing server logs, do not trust the user-agent string alone. Verify the IP and reverse DNS before blaming or blocking the real crawler.

What To Fix If Common Crawl Does Not See You

If the command returns nothing, avoid panic. Start with a boring technical checklist.

1. Check robots.txt: look for User-agent: CCBot or wide AI crawler disallow rules.
2. Check CDN and WAF rules: a clean robots file means little if the edge returns 403 to CCBot-like traffic.
3. Check the canonical host: query both example.com/* and www.example.com/* if your site uses both in redirects or older URLs.
4. Check status codes: redirects can be fine, but widespread 403, 404, 5xx, or tiny 200 responses need investigation.
5. Check internal links and sitemap coverage: a domain can be open but poorly discovered.
6. Check rendered content: if the main content only appears after JavaScript, a crawler may capture a thin shell.
7. Check structured evidence: article, organization, author, product, service, FAQ, breadcrumb, and local data should match visible content.

This is where classic technical SEO still carries the weight. AI systems changed the downstream surfaces, not the need for clean crawl access. I covered the same principle from Google's side in Google's AI search documentation: crawlable, helpful, well-structured pages still do the work.

Turn The Tip Into A Repeatable Audit

I would add this as a small section inside every AI SEO audit:

Pair that with prompt tracking, citation reports, log-file review, and a content structure audit. Prompt tools can tell you whether you appear in answers. Common Crawl tells you whether one of the upstream public crawl layers has seen you at all.

If you are building Claude-based SEO workflows, this is also a good candidate for a small skill: accept a domain, fetch robots.txt, test CCBot access, find the latest collection, query the index, count rows, and return a short pass/fail report. That fits naturally beside a full technical SEO auditor skill for Claude.

FAQ

Does Common Crawl inclusion prove my site was used to train an LLM?

No. It proves Common Crawl recorded URLs from your site in that snapshot. Many AI teams and researchers use Common Crawl data, but the index does not prove that any specific model trained on your specific page.

What does an empty Common Crawl response mean?

It means the query did not find matching rows in that collection. That could mean the site was blocked, not discovered, low priority, queried under the wrong host, missing from that snapshot, or returning content in a way Common Crawl did not capture.

Should I allow CCBot?

That is a business and policy decision. If AI visibility is part of your strategy, blocking Common Crawl can reduce your presence in an important public crawl layer. If licensing, privacy, publisher rights, or legal policy matter more, a block may be intentional. The mistake is not choosing; the mistake is blocking or allowing without knowing.

How often should I run this audit?

Run it after major CDN, WAF, robots, migration, rendering, or platform changes. For active AI SEO programs, a monthly check against the newest Common Crawl collection is a reasonable cadence.

Can this replace AI visibility tools?

No. It checks an upstream crawl layer. AI visibility tools, Bing AI Performance data, referral analysis, server logs, and prompt testing answer different questions. The strongest audit uses them together instead of pretending one surface explains everything.

Final Take

The Common Crawl command is worth adding to your AI SEO toolkit because it gives you something concrete to inspect.

If CCBot is blocked, fix the access decision or document why it is intentional. If the latest index has no rows, investigate discovery, host patterns, status codes, and crawl priority. If the index has rows, do not declare victory. Check whether the important pages are the ones being captured and whether they return useful HTML.

AI SEO is not only prompt screenshots. It starts with whether the machines can reach, store, parse, and reuse the evidence you publish.