Itdaily - Websites can more easily restrict AI training on articles

Websites can more easily restrict AI training on articles

Silver robotic hand pressing a key on a laptop keyboard with one finger, against a blue background.

Google, Anthropic, OpenAI and Microsoft will respect a shared signal that lets websites indicate they do not want AI training on their content. A single setting will then be enough for all participating companies.

Websites will get an easier way to tell AI companies that their texts may not be used to train AI models. Google, Anthropic, OpenAI and Microsoft will respect a common signal for this purpose. A publisher then states its choice once, and every participating AI company reads the same preference.

The initiative is voluntary and builds on more than a year of work by the AI Preferences working group of the Internet Engineering Task Force (IETF). That working group is still working on a definitive standard.

Refusing AI training per bot

Until now, a website administrator had to identify the crawlers of each AI company separately and maintain separate rules for each provider in robots.txt. That is the file in which a site tells crawlers which parts they may visit.

Bot names differ per company, and some companies split training, search and user requests across separate bots. Microsoft, for example, worked with the tags NoArchive and NoCache, for training and for grounding, in which content is used to support generated answers.

Four bots at OpenAI

OpenAI uses four bots, each with its own task. GPTBot collects content that can be used to train OpenAI’s generative AI foundation models. OAI-SearchBot searches websites for the search function of ChatGPT. A site that blocks this bot does not appear in the search answers.

ChatGPT-User visits pages at the request of a user in ChatGPT or a Custom GPT, and the rules in robots.txt may not apply to it. OAI-AdsBot checks pages that have been submitted as an ad in ChatGPT.

Each setting works independently of the others: a site can allow OAI-SearchBot and block GPTBot. OpenAI will keep those settings after the shared signal arrives. The company wants to introduce the signal in the coming months and will then share details about the implementation.

The new signal

So a new signal is coming. The shared signal goes in robots.txt, in the root directory of the website. These two lines refuse AI training on the entire site:

User-Agent: __usage_ai-training
Disallow: /

Anyone who only wants to shield part of a site puts a path after Disallow, for example Disallow: /archief/. If there is no rule, Microsoft assumes permission for now.

Microsoft also reads the preference per page, via a meta tag in the HTML or an HTTP header. That setting takes precedence over robots.txt:

<meta name=”Bingbot” content=”ai-training:disallow”>
X-Robots-Tag: Bingbot: ai-training:disallow

For grounding, only a per-page setting exists. With ai-grounding:disallow, a site excludes grounding entirely. With ai-grounding:disallowextended, a website blocks only the services that Microsoft provides to third parties, such as Web IQ and grounding via Foundry, and Copilot and Microsoft 365 may still use the page.

Microsoft warns that both choices reduce a brand’s visibility in AI applications. A site’s position in Bing search results does not depend on either of these settings.

Microsoft will apply the new preferences from 1 February 2027, each time it crawls a site again. After 1 March 2027, it will no longer read NoArchive and NoCache as a preference for training or grounding. OpenAI will follow next year. Meta has already said it supports the idea, but has not yet communicated about implementation. Anthropic is also said to follow the signal.