For two years, a site owner who wanted to keep content out of AI training had one blunt choice: block the crawler and hope it did not also cost a place in AI answers. On October 30, 2026, that choice was split in two.
Google, Microsoft and OpenAI each published posts the same day backing one shared robots.txt rule for AI training, and Anthropic said it has already started implementing it. Microsoft also added a second, separate preference for "grounding": whether a page may be used to build the answers Copilot and Bing's AI features give. For a B2B company that wants to be named when a buyer asks an AI who to hire, the difference between those two settings is the whole story.
The shared training rule
The rule comes from the IETF's AI Preferences working group. It is a new robots.txt group with a reserved name, __usage_ai-training, and ordinary Allow and Disallow lines under it:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
Two details matter. A wildcard group (User-Agent: *) does not count as a training preference, so a file without a group naming the token says nothing at all. And the token is a use preference, not a crawl directive: it does not stop any bot from fetching a page.
It is also still a draft. The document is an individual Internet-Draft that expires on May 3, 2027, and it has no enforcement mechanism. OpenAI calls its participation voluntary and says it will implement the signal "in the coming months." Meta, Amazon and Apple made no commitment.
Microsoft's second switch
Microsoft's post adds a page-level preference that the shared rule does not cover. A page can now say ai-grounding:disallow in a Bingbot meta tag or an X-Robots-Tag header. There is no site-wide robots.txt version. A middle value, disallowextended, keeps the page in Microsoft's own products (Copilot, Microsoft 365, Bing's AI answers) but out of the grounding services Microsoft sells to other developers.
Microsoft is clear about the trade. The new preferences "do not change eligibility for traditional Bing search results," but choosing Disallow or DisallowExtended for grounding "will reduce brand visibility in AI experiences." That is an engine saying, in its own documentation, that blocking AI use costs you mentions in AI answers.
The dates are spread out. Microsoft starts honoring the new settings on February 1, 2027. It stops reading the old NoArchive and NoCache tags as AI controls after March 1, 2027.
Google's version has a catch
Google says it will honor the shared token too, with "early 2027" as the target in its crawler documentation. But it will treat a training opt-out as if the site had disallowed Google-Extended. And Google-Extended covers more than training: it also covers grounding in the Gemini apps and Vertex AI.
So the same robots.txt line means different things at the two companies. At Microsoft, opting out of training leaves your pages eligible for Copilot's answers. At Google, it takes them out of Gemini's answers as well. A team that adds the shared group to protect its content from training, and assumes that is all it did, could drop out of one of the engines its buyers use.
What the token does not touch at Google is AI Overviews and AI Mode. Those still run on ordinary Googlebot rules, plus a separate Search Console setting.
What we would set
For most B2B startups, the right answer is to change nothing on the grounding side. Your product pages, pricing page, comparison pages and case studies are exactly what you want an answer engine to read when a buyer asks for options. Disallowing grounding on them trades a theoretical protection for a measurable loss.
The training side is a real decision, and it is yours to make. If you do opt out, do it knowing the Google mapping, and check what Gemini says about your category before and after. If a page holds something you do not want quoted, such as a customer list or a draft price sheet, the page-level grounding tag now lets you exclude that page alone instead of the whole site.
The one thing not to do is let the default make the choice. A file with only a wildcard group says nothing, and Bing's temporary settings for sites still on NoArchive or NoCache open in Webmaster Tools on December 15, 2026.
The bigger picture
Every engine makes two decisions about your company: whether it may read your pages, and whether it names you when a buyer asks. Until this week those were tangled together in one robots.txt line. Microsoft has now separated them on paper, and Google has separated them only partly. The second decision is the one that brings in buyers, and it depends on what is on the pages the engines are allowed to read.
ClawWorld runs AI visibility work for B2B startups. Start with a free baseline report: 10 buyer questions, who ChatGPT and Claude name instead of you, and the pages they cite.