Social Rocket
Websites will be able to use a shared `robots.txt` rule to state that their content should not be used to train generative AI. Anthropic, Google, Microsoft and OpenAI announced support for a corresponding proposal 2026 at the end of October. For publishers, brands with knowledge bases and operators of editorial websites, this significantly reduces the work of maintaining individual bot rules.
One switch for several AI providers
The technical draft comes from Anthropic standards specialist Kevin Kelley and was submitted to the Internet Engineering Task Force on 30 October 2026. It uses a special user-agent name in the existing `robots.txt` file without introducing new commands. An example from the draft looks like this[](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-00):
```txt User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/ ```
With this, a website operator declares that content should in principle not be used for AI training; the directory `/press-kit/` is excluded. The rule is a usage preference, not a block for all bots. Search engine crawling can be controlled separately.
Until now, publishers had to manage separate rules for different systems. A shared signal creates a central default that participating providers can take into account alongside their specific controls. The proposal is currently an IETF Internet-Draft and not yet an adopted standard.
Google, Microsoft and OpenAI are not starting at the same time
The companies have different timelines:
- Anthropic says it has already begun rolling this out.
- OpenAI announced on 30 October that it will honor the signal in the coming months. Existing individual crawler rules remain in place.
- Google plans to implement it for 2027. If no explicit Google-Extended rule is set, the shared opt-out should be treated like a corresponding ban on Google-Extended.
- Microsoft names 1 February 2027 as its start date and adds its own settings for AI training and for the use of sources in current answers, so-called grounding. The Bing documentation explains the transition period for older `NoArchive` and `NoCache` signals until 1 March 2027.
So the decisive practical question is not only which rule a website sets, but when individual providers actually process it. Meta has also publicly endorsed a common solution but has not yet committed to a concrete implementation. A good overview of the current commitments is provided by Digiday.
Training, AI answers and search results are three different things
The new mechanism primarily concerns training future models. For marketing teams this is an important distinction: refusing training data does not automatically decide whether a page can be found in search results or whether it is cited in an answer.
Microsoft therefore wants to make training and grounding explicitly controllable independently of each other. Google keeps its existing Google-Extended controls. The separate discussion about opt-out and click data in Google AI Overviews concerns another part of the problem: the visibility of publisher content within generative search products.
A clear rights policy can even be part of a distribution strategy. Some brands want to keep protected content out of training data while still making product knowledge, documentation and press information accessible to search systems.
What publishers and companies should prepare now
Anyone running a larger content platform can start by documenting the existing `robots.txt` rules, planning the new preferences by content type, and keeping an eye on later implementation dates. At Microsoft, operators still using `NoArchive` or `NoCache` for AI purposes should pay particular attention.
For day-to-day operations, the question remains which bots identify themselves correctly. Srocket has already reported on demands for more clearly identifiable AI crawlers. The new rule simplifies communication with cooperating services; controlling the actual access remains a separate task.
Behind the scenes, the disputes over commercial website use also continue. The disputed Meta crawler clauses show why clear responsibilities between technology, editorial and legal teams are becoming ever more important.
The next step is standardisation instead of maintaining a dozen bots: A website operator can formulate its basic decision once and then set targeted exceptions. Whether this also leads to a balanced economic relationship between publishers and AI providers will depend on practical implementation and licensing models.
Image: Danial Igdery / Unsplash.