What changed
On October 30, 2026, Google Search Central and Microsoft Bing announced support for a shared AI-training preference based on an IETF AIPREF Internet-Draft. The proposed robots.txt policy token is __usage_ai-training. Google aims to honour it in 2027, treating a disallow as a Google-Extended disallow unless an explicit Google-Extended rule says otherwise. Bing plans to honour the shared training preference and introduces independent page-level AI Grounding settings from February 1, 2027. Sites using Bing's older NoArchive or NoCache directives to restrict AI uses must migrate by March 1, 2027, including a temporary Bing Webmaster Tools setting for grounding while pages await recrawl.
Why it matters
Site operators have previously needed vendor-specific crawler and content-use controls, and blocking AI training could be confused with losing AI-answer citations. The shared training signal aims to reduce that complexity. Microsoft's separate grounding settings distinguish whether content can inform AI answers in Microsoft products or third-party grounding services, without changing eligibility for ordinary Bing search. Existing NoArchive/NoCache sites face a concrete deadline: without new preferences, grounding may default to Allow after March 1 while pages await recrawl.
The shared training signal is proposed, not yet a final IETF standard
The Internet-Draft uses a reserved robots.txt user-agent-style token, __usage_ai-training. For example, User-Agent: __usage_ai-training followed by Disallow: / signals a site-wide training refusal; path-specific rules can narrow scope. The IETF working group has not finalised the broader preference vocabulary. Google says support is planned for 2027, not already universally enforced.
Google retains explicit Google-Extended overrides
Google says it will interpret the shared disallow as Google-Extended disallow when no explicit Google-Extended preference is set. Publisher-specific Google-Extended instructions continue to work alongside the shared signal. Google's documentation distinguishes AI training controls from the ability to appear in Gemini Apps grounding; do not assume all generative-search citations are governed by one robots.txt directive.
Bing splits training and grounding
Bing will allow AI Training to be set in robots.txt or page-level Bingbot meta/header directives. AI Grounding is page-level only: Allow, Disallow or DisallowExtended. The last option permits Microsoft-operated experiences, including Copilot and Microsoft 365, while excluding third-party grounding services Microsoft provides. These are use preferences, not a change to normal search-indexing eligibility.
Legacy NoArchive and NoCache users must migrate
New Bing preferences begin taking effect on recrawl from February 1, 2027. By March 1, operators relying on NoArchive or NoCache must publish the new directives. To preserve grounding restrictions during recrawl lag, eligible sites also need to register a temporary preference in Bing Webmaster Tools; the temporary control opens December 15, 2026 and expires on recrawl or September 30, 2027. Without that transition, legacy restrictions stop being treated as grounding preferences after March 1.
What the controls do not guarantee
Bing says the defaults for both training and grounding are Allow unless applicable settings say otherwise. Neither platform promises search placement, citations, referral traffic or payment. Robots.txt policy preferences are not a substitute for access controls and depend on providers honouring them. Publishing a page-level setting is not immediate enforcement: crawlers must retrieve the page first.