The New robots.txt AI Training Opt-Out: What It Blocks and What It Doesn't
__usage_ai-training lets one robots.txt group opt out of AI training at Google, OpenAI, Microsoft and Anthropic. What it blocks, what it doesn't, and how to audit.
You open robots.txt to stop AI training. Keeping search visibility takes a more careful edit.
The new robots.txt AI training opt-out uses User-Agent: __usage_ai-training to express a shared training preference. On October 30, 2026, Anthropic, Google, OpenAI and Microsoft backed it; it is neither a crawl block nor a search opt-out. As of October 31, none of the four says it is live for its crawlers.
That distinction matters if your brief is simply “block AI bots.” Training, search indexing and fetching a page for an answer serve different purposes. Decide which uses your business accepts before editing the file.
| Company | What it committed | When | Source |
|---|---|---|---|
| Anthropic | Already started implementing; rollout across services will take time | No live date | IETF mail, Oct 30, 2026 |
| Will treat opt-out as Google-Extended disallow if Google-Extended not specified | 2027 (blog); early 2027 (crawler docs) | Search Central, Oct 30, 2026 | |
| Microsoft | Honors AI Training preference as it crawls; adds page-level training and grounding | Feb 1, 2027; migrate from NoArchive/NoCache by Mar 1, 2027 | Bing blog, Oct 2026 |
| OpenAI | Will honor the common signal | “in the coming months”, no date | OpenAI, Oct 30, 2026 |
What the robots.txt AI training opt-out actually expresses
The individual IETF draft uses ordinary robots.txt groups. The new element is a policy token in the User-Agent line. Its example is:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
This expresses a preference against training for all paths except /press-kit/, where it expresses a preference in favor. It uses existing RFC 9309 path matching and longest-match precedence, without new directives or grammar.
A wildcard group, User-Agent: *, must not match this policy token. No __usage_ai-training group means no preference under this mechanism. Matching is exact apart from case. Consecutive User-Agent lines may combine tokens, including policy and product tokens, in one group.
The draft defines AI training as using an asset to modify a generative model’s learned parameters. It excludes models used exclusively for non-generative tasks; classification, ranking and scoring models fall outside its generative-model definition. It does not describe every possible AI use.
The signal expresses a content-use preference. It does not instruct a crawler to stop requesting pages, and private agreements may override it. Existing proprietary tokens retain their meanings.
This is version 00, an individual submission with intended Standards Track status. It is not IETF-endorsed, has no formal standing and expires May 3, 2027. Treat implementation announcements separately from the draft’s status.
Who committed and when
Google says: “We aim to support this new control along with other AI providers in 2027.” Its crawler documentation narrows that intention to early 2027. The mapping to Google-Extended depends on whether you explicitly specify Google-Extended.
OpenAI says: “We aim to implement the signal in the coming months, and will share another update”. Its announcement also says existing robots.txt controls remain. There is no calendar date to put into a launch checklist yet.
Microsoft gives February 1, 2027 as the start for honoring preferences as it crawls. Its Bing announcement warns: “Neither choice guarantees inclusion, traffic, or payment.” That is a useful limit on both opt-in and opt-out expectations.
The Verge reports that Meta’s Andy Stone signaled support for a common signal. That is not a commitment that Meta will honor this one. Keep Meta outside the four-provider implementation table.
Separate training, search and AI answers
| Control | What it governs | What it does not |
|---|---|---|
| Shared token | Future shared training preference by path | A crawl block or general search opt-out |
| GPTBot | OpenAI training collection preference | OAI-SearchBot or user fetch settings |
| OAI-SearchBot | OpenAI search visibility | GPTBot training preference |
| ChatGPT-User | User-initiated fetches | A replacement for training controls |
| Google-Extended | Google training control; Gemini Apps grounding eligibility | Google Search inclusion or ranking |
| Googlebot | Google Search, including AI Overviews and AI Mode | The separate Google-Extended preference |
| ClaudeBot | Collection that may contribute to training | Claude search and user-fetch settings |
| Claude-SearchBot | Improving Claude search results | ClaudeBot training preference |
| Claude-User | Fetches for user queries | A training-only control |
| Bing AI Training | Training preference, including page-level override | Bing grounding or Search eligibility |
| Bing AI Grounding | Separate page-level grounding preference | Training permission or Search eligibility |
The OpenAI bot documentation makes its settings independent. You can disallow GPTBot while allowing OAI-SearchBot. ChatGPT-User handles user-initiated fetches.
The Anthropic crawler documentation similarly distinguishes ClaudeBot, Claude-SearchBot and Claude-User. Blocking the search or user-fetch bots may reduce the corresponding visibility. A training decision should not accidentally become an answer-discovery decision.
Google is the important overlap. When Google-Extended is unspecified, Google plans to read the shared opt-out as Google-Extended disallow. Explicitly allowing Google-Extended preserves eligibility for grounding responses and clicks in Gemini Apps, according to Google. Eligibility does not establish that your pages will appear.
Google-Extended has no separate HTTP user agent. Google documents it as a control token and says it affects neither Search inclusion nor ranking. For Google AI Overviews and AI Mode, see the search-side control. That is a separate decision from this training preference.
Bing separates training from grounding, with both defaulting to Allow. Page-level training overrides robots.txt. Grounding supports Allow, Disallow and DisallowExtended; the last limits grounding to Microsoft-operated experiences such as Copilot, excluding third-party grounding services such as Web IQ and Foundry API.
Bing Search eligibility is unaffected. Microsoft says blocking grounding reduces brand visibility in AI experiences. Replace NoArchive/NoCache as training and grounding preferences by March 1, 2027: after that, Microsoft stops reading them for those purposes.
For sites using those legacy controls, a temporary grounding preference in Bing Webmaster Tools becomes available December 15, 2026. It lasts until recrawl or September 30, 2027. Record both the migration and the recrawl dependency.
Expert Insight I would treat training permission and AI answer visibility as two business decisions. The error I expect most is blocking search and user-fetch bots alongside training bots. Keep the reason for each rule beside the change request.
Common robots.txt mistakes and their fixes
These are my reading of the documented controls, not measured error rates.
- One broad block for every AI agent. Fix: review GPTBot and ClaudeBot separately from search and user-fetch agents. Match each rule to the use you intend to restrict.
- Assuming the wildcard includes the policy token. Fix: add an explicit
__usage_ai-traininggroup when you want to express this preference. - Leaving Google-Extended undecided. Fix: explicitly decide whether its training and Gemini grounding scope is acceptable. Review Google’s planned fallback before publishing.
- Expecting requests or previous training to disappear. Fix: distinguish future use preferences from crawl behavior. No source says the shared token applies retroactively.
- Blocking Bing grounding to stop training. Fix: configure the independent training preference. Restrict grounding only if its visibility tradeoff is intended.
- Relying only on NoArchive/NoCache. Fix: migrate Bing preferences by March 1, 2027, and check the page-level settings.
- Treating robots.txt as security. Fix: do not rely on it to prevent access. Non-compliant scrapers can ignore it.
Keep your policy decisions with the audit notes. Our AI crawlers and llms.txt coverage provides the broader topic context without making these controls equivalent.
Audit checklist: decide, edit, verify
# Example: shared training preference, with current vendor controls.
User-Agent: __usage_ai-training
Disallow: /
User-Agent: GPTBot
Disallow: /
User-Agent: ClaudeBot
Disallow: /
# Explicit choice. Disallow keeps training off at Google.
# Allow would keep Gemini Apps grounding eligibility, but also allows training.
User-Agent: Google-Extended
Disallow: /
User-Agent: OAI-SearchBot
Allow: /
User-Agent: Claude-SearchBot
Allow: /
User-Agent: ChatGPT-User
Allow: /
User-Agent: Claude-User
Allow: /
This example disallows Google-Extended explicitly, so the Google decision is visible instead of implied. The cost is Gemini Apps grounding eligibility; if those answers matter more to you than the training preference at Google, change that one group to Allow and write down why. “Training off everywhere, answers on everywhere” is not an option Google offers through this token.
Review existing Googlebot and Bingbot access separately. Bing’s page-level grounding preferences also need their own review; this snippet does not configure them.
Measure GSC and AI referrals without inventing a result
Baseline now: none of the four has announced the signal as live yet. The token alone should not change Google Search rankings; Google-Extended is not a ranking signal. A ranking movement after editing robots.txt would not establish this token as the cause.
Use the Search Console Generative AI performance report for impressions by page, country, date and device. It does not supply clicks. Keep that limitation visible when comparing exposure with analytics sessions.
Segment referrals from chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com and claude.ai. Compare four weeks before and after a change. That comparison describes observed sessions; it does not isolate the effect of this preference.
Compare server-log hits for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Googlebot and Bingbot. The policy token will not change crawling by itself. Check Bing Webmaster Tools AI performance or citations where available.
Annotate October 30, your publication date, February 1, 2027 and Google’s early-2027 target. Keep announced dates separate from verified implementation. Search exposure, referrals and crawler requests answer different questions.
FAQ
Does __usage_ai-training block AI crawlers?
No. It expresses a training preference about content use. Keep provider-specific crawl controls where those match your policy.
Does it affect Google Search or AI Overviews?
It is not a Search opt-out. Google-Extended affects neither Search inclusion nor ranking; Google’s AI search controls are separate.
Does User-Agent: * cover it?
No. The draft explicitly excludes wildcard matching for this policy token. Without its own group, this mechanism expresses no preference.
Do I still need GPTBot and Google-Extended rules?
Existing controls remain valid. Keep deliberate provider rules, particularly while shared support is pending, and make Google-Extended’s grounding tradeoff explicit.
When does it start working?
No provider says it is live as of October 31, 2026. Microsoft gives February 1, 2027; Google targets 2027, with early 2027 in its docs. OpenAI says “in the coming months”; Anthropic gives no live date.
Does it remove content already used for training?
No source says it applies retroactively. Do not describe publishing this group as removing prior training material.
A published preference is not proof of implementation or compliance.
Growing Search can help review these choices alongside your search visibility in an SEO audit.
About the Author
