A shared way for websites to express AI training preferences is gaining support from some of the largest AI companies. On October 30, 2026, OpenAI, Google, and Microsoft each announced plans to honor a common signal in robots.txt. The proposal uses a policy token called __usage_ai-training, allowing a website operator to communicate a preference about whether its content may be used to train generative AI models without maintaining a separate training rule for every participating provider.

Anthropic is closely involved in the proposal: the Internet-Draft was authored by Kevin Kelley, whose listed affiliation is Anthropic, and The Verge reported that Anthropic has begun implementing the signal. However, the document is still an Internet-Draft, not a finalized IETF standard. The companies also have different implementation timelines, so website owners should not assume that the new control is already active everywhere.

The announcement matters to publishers, bloggers, businesses, and website administrators because it addresses a growing question: Can a website allow AI systems to discover and cite its pages while expressing a separate preference about using those pages to train generative models?

What did OpenAI, Google, Microsoft, and Anthropic announce?

The shared proposal is designed to make AI training preferences easier to express across participating services. Instead of adding a separate training preference for every AI provider, a publisher can use one policy token in a standard robots.txt group.

The announcements are related, but their status and timing are not identical:

  • OpenAI: announced on October 30 that it will honor the common signal, aiming to implement it in the coming months. OpenAI says its existing crawler controls will remain in place.
  • Google: announced plans to support the shared AI training control in 2027. Google says the shared opt-out will be interpreted as a Google-Extended disallow when no explicit Google-Extended instruction overrides it.
  • Microsoft: announced support for the shared training preference and additional, separate controls for AI training and AI grounding. Microsoft says it will begin honoring the new preferences as it crawls from February 1, 2027.
  • Anthropic: an Anthropic-affiliated author published the IETF Internet-Draft, and The Verge reported that Anthropic has begun implementing the signal. The draft’s author affiliation should not be confused with formal IETF approval or a completed industry-wide rollout.

This is a coordinated move toward a common mechanism, not proof that every AI company or crawler already recognizes the token.

What is the new AI training signal in robots.txt?

The proposed policy token is:

__usage_ai-training

It appears in the User-agent line of a normal robots.txt group. A website that wants to express a preference against using all its paths for generative AI training could use the following example:

User-agent: __usage_ai-training
Disallow: /

A publisher could also make a path-specific exception. For example, the following illustrative rules express a preference against training on most of the site while allowing the /press-kit/ path:

User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

These examples follow the October 30 IETF Internet-Draft. They explain the proposed syntax; they are not a recommendation to add the rule to every website immediately. Providers must implement the signal before it can be relied on, and existing provider-specific controls may still be necessary.

The draft also says that a wildcard group such as User-agent: * does not automatically express a preference for this policy token. If a website has no group naming __usage_ai-training, the draft treats that as no preference expressed through this mechanism.

Does the new rule block AI crawlers from visiting a website?

No. This is one of the most important distinctions in the proposal.

The shared token communicates a preference about AI training use. It is not, by itself, a crawler-access directive. A crawler may still be permitted to fetch a page under its own robots.txt rules even when the publisher has expressed a preference against using that content to train a generative model.

Website owners therefore need to distinguish three separate activities:

  1. Crawling: a bot requests and retrieves a page from a website.
  2. AI training: content is used to modify the learned parameters of a generative AI model.
  3. AI search or grounding: content is retrieved or used to help produce an answer, potentially with a citation or link to the original page.

The IETF draft explicitly separates content-usage preferences from crawler traffic-management rules. If a publisher wants to restrict a particular crawler from accessing pages, the publisher still needs the relevant crawler-specific rules. For OpenAI, for example, the search crawler and training crawler have separate purposes and controls.

For a practical explanation of how AI-powered web search works, see our guide to How ChatGPT Search Works: Sources, Citations, and Limits.

Will opting out of AI training remove a website from search results?

The shared training preference is not designed to act as a general search de-indexing instruction. Google says its existing company-specific controls will continue to work and explains that the shared training opt-out can be interpreted as a Google-Extended disallow. Google also says that websites explicitly allowing Google-Extended remain eligible for grounding in Gemini Apps.

Microsoft makes a similar distinction in its October 30 announcement: its new AI training and AI grounding preferences are separate from eligibility for traditional Bing search results. Microsoft says the new preferences do not change a site’s eligibility for ordinary Bing results, subject to normal crawling and indexing requirements.

OpenAI also distinguishes its search crawler, OAI-SearchBot, from GPTBot, which is associated with potential model training. OpenAI’s crawler documentation explains that publishers can allow search discovery while disallowing the training crawler. That distinction is important for sites that want to be discoverable in ChatGPT Search without making the same choice about training.

However, being eligible to appear is not a guarantee of being cited, ranked, or sent traffic. Search systems decide which sources are relevant to each query, and visibility depends on more than a robots.txt setting.

Website owners interested in Google’s AI search experience can also read our guide on How to Use Google AI Search to Find Sources and Official Websites.

Microsoft adds a separate AI grounding preference

Microsoft’s announcement goes beyond the shared training signal. It introduces distinct choices for AI training and AI grounding.

In Microsoft’s terminology, grounding means using website content to help inform AI-generated responses. This can include experiences such as Copilot and Microsoft 365. Microsoft says publishers will be able to express grounding preferences at page level, separately from the shared training preference.

The company describes three grounding choices:

  • Allow: the page may be used to help inform generated responses. This is the default when no applicable preference is set.
  • Disallow: the page should not be used for grounding.
  • DisallowExtended: the page may be used in Microsoft-operated experiences but excluded from certain grounding services Microsoft provides to third parties.

Microsoft says the new preferences will be honored as it crawls beginning February 1, 2027. It also says that choosing to disallow grounding can reduce a brand’s visibility in AI-generated experiences, even though traditional Bing search eligibility is unaffected.

This distinction matters to publishers. A site may want to prevent its articles from being used to train future models while still allowing those articles to inform answers that link back to the publisher. Training and grounding are different permissions, and the new proposal makes that difference more explicit.

Microsoft Bing implementation: exact dates and migration steps

Microsoft’s October 30 announcement includes a transition plan for publishers that currently rely on the legacy NoArchive or NoCache directives. The dates are important because the new preferences do not become effective immediately when announced.

Date (2026–2027) What happens
December 15, 2026 Temporary AI Grounding transition preferences open in Bing Webmaster Tools. This applies only to sites using NoArchive or NoCache as of that date.
February 1, 2027 Microsoft begins honoring the new AI Training and AI Grounding preferences as it crawls pages.
March 1, 2027 Deadline for publishers relying on the old directives to publish the new preferences and, where required, record a temporary grounding preference in Bing Webmaster Tools.
September 30, 2027 Temporary grounding preferences expire, unless Microsoft has already recrawled the page and discovered its new page-level preference.

If your site does not use NoArchive or NoCache, Microsoft says no temporary transition action is required. The new controls still need to be published if you want to express a preference other than the default Allow.

Which settings go where?

Microsoft’s controls are not all configured in the same place:

  • AI Training: can be expressed site-wide or for a path in robots.txt using the shared __usage_ai-training token. Microsoft also supports a page-level override using a meta tag or HTTP response header.
  • AI Grounding: is page-level only. It cannot be set in robots.txt. Use a page meta tag or HTTP response header to choose Allow, DisallowExtended, or Disallow.
  • Page-level precedence: a page-level AI Training preference overrides the applicable robots.txt preference for that page.
  • Legacy directives: from February 1, 2027, Microsoft will disregard NoArchive and NoCache for the corresponding training and grounding preferences when it detects the new controls. After March 1, those legacy directives will no longer be read as preferences for AI training or grounding.

For publishers who need to disallow AI training on a specific page, Microsoft’s documented meta-tag example is:

<meta name="Bingbot" content="ai-training:disallow">

To disallow AI grounding on a page, the documented form is:

<meta name="Bingbot" content="ai-grounding:disallow">

A more limited grounding choice can use:

<meta name="Bingbot" content="ai-grounding:disallowextended">

The equivalent page-level preferences can also be sent through an X-Robots-Tag HTTP response header. Follow Microsoft’s current implementation guide and FAQ before deploying these examples across a production site.

Important risk for sites using NoArchive or NoCache

If a site currently uses NoArchive or NoCache and wants to continue restricting grounding while Bing recrawls its pages, Microsoft says the publisher must both publish the new page-level AI Grounding preference and record the temporary grounding preference in Bing Webmaster Tools before March 1, 2027. If the temporary preference is not recorded, pages using the old directives may default to Allow for grounding until Microsoft recrawls them and discovers the new page-level preference.

The temporary setting is a migration safeguard for grounding; Microsoft says there is no equivalent temporary Bing Webmaster Tools setting for AI Training. Training preferences must be published in robots.txt or at page level and will take effect as Microsoft crawls.

Official implementation details: Microsoft Bing announcement and implementation guide and Bing Webmaster Tools FAQ: New AI Preferences.

When will the shared AI training control take effect?

The companies have announced different timelines:

Provider or organization Announced status
OpenAI Aims to implement the shared signal in the coming months
Google Aims to support the control in 2027
Microsoft Begins honoring new preferences from February 1, 2027; transition settings open December 15, 2026 for eligible legacy-directive users; migration deadline March 1, 2027
Anthropic The draft is authored by an Anthropic-affiliated contributor; The Verge reports implementation has begun
IETF The proposal is an individual Internet-Draft, not an approved RFC or finalized standard

These dates are implementation targets, not a guarantee that every page will be processed on a specific day. A provider must crawl or otherwise process the relevant instructions before a newly published preference can affect its systems.

The IETF Datatracker states that the draft is an individual Internet-Draft and is not endorsed by the IETF or formally adopted in its standards process. The draft is therefore a proposal under discussion, and its wording or implementation details may change.

What should website owners do now?

For most publishers, the sensible first step is to understand the existing crawler rules rather than immediately changing them.

  1. Review the current robots.txt file. Check which crawlers are allowed or disallowed and make sure important site pages are not blocked unintentionally.
  2. Separate search access from training preferences. Do not assume that allowing a search crawler automatically means agreeing to AI training, or that blocking a training crawler removes a site from search.
  3. Check each provider’s official documentation. OpenAI, Google, and Microsoft have different existing controls and rollout schedules.
  4. Wait for implementation details where necessary. OpenAI says it will provide another update when its implementation is ready; Google targets 2027, and Microsoft has announced February 1, 2027 as its start date.
  5. Test after making any change. Confirm that the live robots.txt file contains the intended rules and monitor provider documentation for changes.

Do not copy the illustrative Disallow: / example without deciding what it means for your publishing strategy. It expresses a preference against AI training for the entire site under this policy token; it does not simply improve search visibility or block all bots.

For website administrators troubleshooting automated access, our guide to Fixing ChatGPT Cloud Browser Website Access Problems explains why website access rules and automated browsing permissions can matter.

What the announcement does not mean

Several claims would go beyond what the announcements establish:

  • It does not mean the IETF has finalized a universal AI content-control standard.
  • It does not mean all AI companies already honor the new token.
  • It does not guarantee that a website will appear in ChatGPT, Gemini, Copilot, or other AI answers.
  • It does not create a payment or licensing agreement between AI companies and publishers.
  • It does not automatically prevent crawling, copying, or use of content by services that do not implement the signal.
  • It does not retroactively remove material that may already have been used in model training.

The proposal is a machine-readable way to communicate a publisher’s preference to participating providers. It is not a technical enforcement system that can independently prevent every possible use of publicly accessible content.

Why this matters for the future of the open web

The shared AI training preference addresses a practical problem for website operators: maintaining different rules for a growing number of AI providers is difficult, especially for publishers managing large sites. A common signal could reduce that complexity and make training preferences easier to communicate.

The separation between training and grounding is equally significant. Publishers may want their reporting, tutorials, and research to remain discoverable and citable while retaining a distinct preference about whether the same material can be used to train generative models. Clearer controls can make those choices easier to express, although the value of the system will depend on broad adoption and transparent implementation.

For independent websites, the key is to preserve the distinction between being crawled, being included in search, being cited in an AI answer, and being used for model training. Those are related but separate processes. Understanding them will help publishers make informed decisions as AI search and content-use policies continue to evolve.

Official sources and further reading

Reporting note: This article is based on company announcements, Microsoft’s implementation guide and FAQ, and the IETF draft published on October 30, 2026. The shared IETF proposal remains under development; Microsoft’s separate publisher controls and migration dates are described according to its official documentation. Last checked: October 31, 2026.