On 30 October 2026, three of the biggest AI companies announced support for one small thing: a single line in robots.txt that tells them not to train on a site. A fourth, Anthropic, is reported to be building support through the engineer who wrote the specification. The line is a policy token called __usage_ai-training, defined in an Internet-Draft by Anthropic’s Kevin Kelley. The announcement produced four different timelines, and no company says it fully honours the token yet. A site that adds it today is writing a preference whose readers arrive on four different schedules, under a document that expires in May 2027.
The appeal is obvious. Until now, opting out of AI training meant a separate rule for every company. The new token collapses that into one group that participating providers have agreed to honour. The catch is just as clear: the token is a draft, it covers training only, and it does nothing on its own until each company switches it on.
What is the __usage_ai-training token?
The mechanism is defined in the Internet-Draft “Machine-Readable Training Preference Signal”, dated 30 October 2026, sole author Kevin Kelley of Anthropic. It introduces no new directive names and no new grammar. It reuses the ordinary robots.txt group structure, with the token sitting where a crawler name normally goes. A site that wants out of training writes this:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That group expresses a preference against AI training for everything except the pages under /press-kit/, which it opts back in. The token must match exactly, apart from letter case, and the double-underscore prefix makes accidental matches unlikely.
One trap matters more than any other. A wildcard group never expresses a preference under this draft. The draft states that wildcard groups, meaning User-Agent: *, must not be interpreted as matching the policy token, so a site that blocks every bot with a wildcard rule has said nothing about AI training. The token has to be named explicitly, or no preference exists at all.
What the token does not do
The token is not a crawl rule. The draft states plainly that it is not a directive for crawling behavior, and that content-usage preferences and crawler signals are independent of each other. Adding the token does not stop any bot from visiting a page, and it does not change search inclusion or ranking.
It does not govern AI answers that quote pages at answer time. Training means using content to modify the learned parameters of a generative model. Retrieval, where a system pulls a live page into an answer, changes no model parameters, so it falls outside the token. Microsoft treats that retrieval step as a separate choice; Google bundles it into its existing control, as explained below.
It is not retroactive. The preferences that apply to a page are those in the robots.txt file in effect when the page was retrieved. The token does not remove content already collected into datasets or already trained into models.
Finally, it is not a standard and not a law. The draft is an individual submission, not a working-group document, and its own boilerplate calls Internet-Drafts working documents valid for a maximum of six months. The draft expires on 3 May 2027 unless it is revised or adopted. OpenAI calls its own participation a voluntary effort, and private agreements may override the preference.
When does each company start reading it?
Three companies announced on the same day, a fourth is reported to be implementing, and each is on a different schedule. No company says it fully honours the token yet, and there is no shared live date. Our compiled timeline, from the verified dates:
| Milestone | Date |
|---|---|
| Draft published; Google, OpenAI and Microsoft posts; Kelley mailing-list post | 30 October 2026 |
| Anthropic implementing, no live date (reported, not company-announced) | Undated |
| OpenAI implementation | In the coming months |
| Microsoft temporary grounding settings open in Bing Webmaster Tools, only for sites using NoArchive or NoCache as of that date | 15 December 2026 |
| Microsoft starts honouring the new preferences as it crawls | 1 February 2027 |
| Google support | In 2027 |
| After 1 March 2027, NoArchive and NoCache no longer read as training or grounding preferences | 1 March 2027 |
| Draft expires unless revised or adopted | 3 May 2027 |
| Microsoft temporary grounding preferences expire, or earlier when the page is recrawled | 30 September 2027 |
Two pieces of arithmetic, calculated by us from those dates, show the shape of the rollout. Microsoft’s 1 February 2027 start falls 94 days after the 30 October announcement, while the draft itself expires 185 days after announcement. The draft’s first version therefore lapses less than four months after Microsoft’s start date unless it is revised or adopted. A publisher adding the token now is early, not late, and the preference only gains readers through 2027.
What each company actually said
Google announced on 30 October, in a post by Search Advocate John Mueller, that it aims to support the shared control along with other AI providers in 2027, building on Google-Extended, the AI training control it has offered since 2023. Google’s interpretation rule is specific: when a site uses the shared token to opt out and does not explicitly mention Google-Extended, Google plans to interpret that as if the publisher had disallowed Google-Extended. Existing company-specific controls continue alongside the shared signal.
OpenAI announced the same day that it will honour a common signal in robots.txt for foundational generative AI training, calling the effort voluntary and saying it aims to implement the signal in the coming months, with details to follow. Its existing robots.txt controls will continue to apply and be respected. One open question: OpenAI’s post does not say whether “foundational” narrows the draft’s definition of training, so the article does not resolve it.
Anthropic’s position is reported, not company-announced. Draft author Kevin Kelley wrote on the IETF ai-control mailing list on 30 October that Anthropic has already started implementing support for the draft, adding that it will take time to add support across its services and that no live date was given. No Anthropic company announcement about the token was found as of 31 October, so every Anthropic claim here carries that reported status.
Microsoft’s plan is the most detailed and the only one with a transition calendar, but its blog post could not be read directly for this article, so its details below are attributed to coverage that quoted the post directly, namely PPC Land and Search Engine Roundtable. Microsoft starts honouring the new preferences as it crawls on 1 February 2027. It adds a page-level training control, set with a meta tag or HTTP header, which overrides the robots.txt setting. It also adds a separate page-level grounding preference with three values: Allow, Disallow, and DisallowExtended, which limits grounding to Microsoft’s own experiences and excludes the grounding services it sells to third parties. Temporary grounding settings open in Bing Webmaster Tools on 15 December 2026 for sites using NoArchive or NoCache, and after 1 March 2027 those older tags will no longer be read as a training or grounding preference. Temporary grounding preferences expire on 30 September 2027, or earlier when the page is recrawled and the new page-level grounding preference applies. Microsoft’s post adds that the preferences do not establish licensing or payment arrangements.
Meta welcomed the idea of a common signal but has not committed to implementing it, according to coverage citing SiliconANGLE, so it is not listed among the companies honouring the token. Apple and Amazon made no 30 October commitment in any source read for this article.
The same token does different things at Google and Microsoft
This is the part a publisher most needs to understand before copying the two lines. At Google, the shared token maps onto Google-Extended, and Google-Extended covers both training and grounding in Gemini Apps and Vertex AI. A site that disallows only the shared token at Google would, under Google’s stated plan, also withdraw from Gemini grounding, because Google reads the token as a Google-Extended disallow. Google’s post does not address that case directly, so it is presented here as our reading of its two documents together, not as a Google quote.
At Microsoft, the same token refuses training only. Grounding is a separate page-level choice. A publisher that wants no training but wants its pages still citable in AI answers has that combination at Microsoft and no single Google setting for it. The Microsoft half of that comparison rests on the secondary coverage cited above.
There is a second trap inside Microsoft’s transition. If Microsoft detects new training or grounding preferences on a site, it will disregard any NoArchive and NoCache preferences that remain on that site. A site relying on NoCache to stay out of Copilot answers that adds only the shared training token would therefore lose that grounding restriction unless it also publishes page-level grounding preferences. The temporary grounding setting in Bing Webmaster Tools is the bridge during the transition: it holds the old restriction until the page is recrawled under the new preferences.
How do I add the token to my robots.txt?
The steps below assume a site that can edit its own robots.txt file. Each step is one action, and the whole job takes minutes.
- Open the robots.txt file at the root of your site for editing.
- Add a new group starting with the line User-Agent: __usage_ai-training.
- Add Disallow: / on the next line to refuse training on everything.
- Add Allow lines for any folders you want to opt back in.
- Do not delete existing per-company rules when you add the shared group.
- Check the file renders with no typos in the token spelling.
- Publish the file and confirm it loads at your robots.txt address.
If you do nothing, nothing breaks. Your existing per-company rules keep their current meaning, and the draft changes nothing on its own. The token is a preference you may express, not a requirement, and no company has announced any penalty for ignoring it. Waiting until each company’s date arrives costs nothing except the head start of having the preference already in place when readers switch on.
Should you add it now?
Adding the token now costs two lines and commits a site to nothing irreversible, since private agreements may override it and the draft itself may change before May 2027. The realistic expectation is patience: nothing changes at any company until its own date arrives, and the earliest firm date is Microsoft’s 1 February 2027. Sites that depend on NoArchive or NoCache at Microsoft should treat 1 March 2027 as their real deadline, not the token’s announcement date.
The honest summary is narrower than the headlines. Four companies are behind one line for refusing AI training, three of them by their own announcements and one reportedly through its draft author. The line is training-only, voluntary, and not yet live as a finished control at any company. What it replaces is not a single old system but a pile of per-company rules, and what it cannot do, blocking crawls, blocking AI answers, or undoing past training, is exactly what many readers will assume it does. Add it for what it is, and do not expect it to do the rest.
