Microsoft said on 30 October that from 1 February 2027 it will honor a shared robots.txt token, __usage_ai-training, that tells AI companies whether a site wants its content used to train models [1]. The token comes from an Internet-Draft filed the same day by Kevin Kelley of Anthropic [2][3]. Microsoft also added a separate, page-level control for grounding in Copilot, Microsoft 365 and Bing, and set 1 March 2027 as the date its older NoArchive and NoCache tags stop counting for either use [1].
At a glance
| Shared token | __usage_ai-training, set in an ordinary robots.txt User-Agent group [2] |
|---|---|
| Microsoft start date | 1 February 2027, applied as pages are crawled [1] |
| Old tags retired | NoArchive and NoCache no longer read for training or grounding after 1 March 2027 [1] |
| Grounding options | Allow (default), DisallowExtended, Disallow; page level only [1] |
| Standards status | Individual IETF draft, not endorsed by the IETF; expires 3 May 2027 [2] |
| Other backers | OpenAI and Google said they will support it, according to several reports [4][5][6][7] |
The signal
- The token sits in a normal robots.txt group, such as User-Agent: __usage_ai-training followed by Disallow: /, and adds no new grammar [2].
- Wildcard groups (User-Agent: *) are never read as a training preference, so today's robots.txt files express none until a site adds the token [2][3].
- Microsoft's grounding control has three settings; DisallowExtended keeps content in Microsoft's own AI products but out of grounding it sells to third parties [1].
- Microsoft says choosing Disallow or DisallowExtended for grounding will reduce brand visibility in AI experiences [1].
- Kelley wrote on the IETF mailing list that Anthropic has already started implementing support for the draft [3].
- OpenAI aims to implement the signal in the coming months, Unite.AI, CellCog and AI Weekly reported; Shopifreaks and CellCog said Google plans support in early 2027 [4][5][6][7].
What happened
Three documents landed on 30 October. Kevin Kelley of Anthropic submitted an Internet-Draft called Machine-Readable Training Preference Signal, and posted it to the IETF ai-control list at 15:39 UTC [2][3]. The draft defines one policy token, __usage_ai-training, that a site writes into the User-Agent line of a robots.txt group. A Disallow rule under that token states a preference against AI training for those paths; an Allow rule states a preference in favour [2]. The draft says the token "is not a directive for crawling behavior", and it is not a standard: the IETF datatracker marks it as not endorsed and with no formal standing [2].
Microsoft's version. On the same day Microsoft's Bing team published its own plan [1]. It will honor the shared training token from 1 February 2027, and will also accept the training preference at page level through a meta tag or an X-Robots-Tag header, where the page setting overrides robots.txt [1]. Separately, it introduced an AI Grounding preference, which covers using a page as input to an AI answer in Bing, Copilot and Microsoft 365 [1]. Grounding can only be set page by page; there is no site-wide robots.txt version [1]. Both preferences default to Allow, and Microsoft says neither affects eligibility for Bing search results [1].
The deadline. The new settings replace NoArchive and NoCache, which Microsoft introduced in 2023 for the same purpose [1]. Sites still relying on those tags have until 1 March 2027 to publish the new preferences. Sites that want to keep content out of grounding must also record a temporary preference in Bing Webmaster Tools, which opens on 15 December 2026 and lapses at recrawl or on 30 September 2027 [1].
Other companies. OpenAI said it will honor the signal and aims to implement it in the coming months, according to Unite.AI, CellCog and AI Weekly [4][5][7]. Shopifreaks and CellCog reported that Google plans to support it in early 2027 [4][6]. These company posts could not be read directly for this briefing, so their timing is attributed to the reports.
By the numbers
Timeline
Microsoft offers NoArchive and NoCache as publisher controls for AI training and grounding [1].
Kevin Kelley of Anthropic submits draft-kelley-aipref-training-preference-signal-00 to the IETF [2][3].
Microsoft announces support for the shared training token and a separate AI Grounding preference [1].
Temporary grounding settings open in Bing Webmaster Tools for sites using NoArchive or NoCache [1].
Microsoft begins honoring AI Training and AI Grounding preferences as it crawls [1].
NoArchive and NoCache stop being read as training or grounding preferences [1].
The Internet-Draft expires unless it is revised or adopted [2].
Temporary grounding settings in Bing Webmaster Tools end [1].
“If you select DisallowExtended or Disallow for grounding, you will see reduced brand visibility in AI experiences.”
How a site sets its AI preferences under the new rules
- 1Decide on training
Add a User-Agent: __usage_ai-training group to robots.txt with Allow or Disallow rules [2].
- 2Decide on grounding
On Bing, set ai-grounding per page with a meta tag or X-Robots-Tag; Allow is the default [1].
- 3Move off old tags
Replace NoArchive and NoCache before 1 March 2027 [1].
- 4Bridge the gap
Sites keeping content out of grounding record a temporary setting in Bing Webmaster Tools from 15 December 2026 [1].
Why it matters
Until now, a company that wanted to keep its pages out of AI training had to learn a separate crawler name and rule for each AI firm. A single token cuts that to one robots.txt group, if the firms that say they will honor it follow through [2][4]. That makes the training decision cheap to act on, and it is likely to come up in legal and brand meetings soon.
The harder choice is grounding, and Microsoft has put it in writing. Training shapes future models; grounding decides whether a page can be quoted, linked and cited in an answer a buyer reads today in Copilot or Bing [1]. Microsoft's own post warns that blocking grounding reduces brand visibility in AI experiences, and that allowing it guarantees neither inclusion, traffic nor payment [1]. A B2B vendor that blocks both because they look alike could drop out of the answers its buyers ask for, while gaining little protection that the training switch alone would not give.
There is also a quiet trap in the defaults. Under the draft, a robots.txt file without the new token expresses no preference at all, and a User-Agent: * block is never read as one [2][3]. Under Microsoft's rules, a site that used NoArchive or NoCache to limit AI use and does nothing by 1 March 2027 falls back to Allow for grounding until its pages are recrawled with new settings [1]. Teams that set these tags years ago, often without marketing in the room, now need to decide again on purpose.
What to do now
For marketers
- Open your robots.txt and page templates and list every NoArchive, NoCache and AI crawler rule already in place.
- Agree with legal on two separate answers: one for AI training, one for appearing in AI answers.
- If you want to be cited in Copilot and Bing answers, leave grounding at Allow and say so in writing.
- Put 15 December 2026 and 1 March 2027 in the team calendar if your site uses NoArchive or NoCache.
For software buyers and IT teams
- When an AI answer cites few vendors in a category, remember some sites may have opted out of grounding.
- Ask vendors whether their documentation is allowed in AI answers, so your team's assistants can read it.
- Check claims in AI answers against the cited page before shortlisting.
Printable, one page, 11 checks. Enter your email and it opens right away.
We email you a copy. We never share your email and never send you sales emails.
Pick one. It takes 5 seconds and helps more people find these briefings. Your checklist opens straight after.
Have a work email? Use it instead and the checklist opens straight away.
Open it, then save it as a PDF or print it.
Know someone who would find this useful?
These steps are our recommendations based on this news, not statements from the companies involved.
What to watch
- OpenAI's details. OpenAI has said it will share implementation details later; whether it also separates training from answer use, as Microsoft did, is not yet known [5].
- Google's mapping. Shopifreaks reported that Google plans to treat a token opt-out as a block on Google-Extended unless a site names Google-Extended directly [6]. Watch for Google's own documentation of that rule.
- The IETF path. The draft is an individual submission that expires on 3 May 2027; adoption by the AIPREF working group would give it formal standing [2][3].
- Grounding elsewhere. No company other than Microsoft has yet published a separate grounding control based on these reports [1][4].
Who is affected
| Who | What changes | Impact |
|---|---|---|
| B2B content and SEO teams | Blocking grounding cuts brand visibility in Microsoft AI answers, so it needs a deliberate decision [1]. | High |
| Sites using NoArchive or NoCache | Must publish new preferences by 1 March 2027 or fall back to Allow for grounding [1]. | High |
| Legal and brand owners | One robots.txt group can now state a training preference to several AI firms [2][4]. | Medium |
Questions readers ask
What is __usage_ai-training in robots.txt?
It is a policy token placed in a robots.txt User-Agent line to state whether a site allows AI training on its content. It was defined in an IETF draft filed on 30 October 2026 [2].
How do I block AI training with robots.txt?
Add a group with User-Agent: __usage_ai-training and Disallow: / for the whole site, or Disallow with a path for one section. Microsoft gives this example in its guide [1][2].
Does blocking AI training remove my site from Bing search?
No. Microsoft says the training and grounding preferences do not change eligibility for Bing search results [1].
What is AI grounding in Bing and Copilot?
Grounding is using a page as input to help write an AI answer, with links and citations. Microsoft lets sites set it page by page, separately from training [1].
What does DisallowExtended mean?
It limits grounding to Microsoft's own products, such as Copilot and Microsoft 365, and excludes grounding services Microsoft provides to third parties through its APIs [1].
When do NoArchive and NoCache stop working for AI?
After 1 March 2027 Microsoft will no longer read NoArchive or NoCache as preferences for training or grounding. The new preferences take effect from 1 February 2027 [1].
Companies in this story
Sources
- Microsoft BingOriginal
- IETF Datatracker
- IETF Mail Archive
- CellCog
- Unite.AI
- Shopifreaks
- AI Weekly
- The Tech Buzz
Researched, fact-checked and edited by the Market Signal desk. Read our editorial standards. Spotted an error? Email hello@webvisiblity.com and we will correct it.