Four AI Giants Adopt a Shared robots.txt Token: Publishers Can Opt Out of AI Training With Two Lines

The four companies that operate the most heavily used AI crawlers on the web have agreed to read from the same page of robots.txt. On October 30, OpenAI, Google, Microsoft and Anthropic each announced that they will honor a shared training-preference signal, __usage_ai-training, which lets a website declare its AI training opt-out once and have it apply to every participating provider, instead of maintaining separate rules for GPTBot, Google-Extended, ClaudeBot and each new crawler that appears.

The signal was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as an Internet-Draft in the Internet Engineering Task Force's AI Preferences (AIPREF) working group, titled "Machine-Readable Training Preference Signal." In practice it is two lines in robots.txt: a User-Agent line naming __usage_ai-training and a Disallow rule, with ordinary Allow lines available to carve out exceptions for specific directories. The draft was published on October 30, 2026 and will expire on May 3, 2027 unless renewed. It remains a draft, not a formal IETF standard, and it binds only the providers that choose to follow it.

Timelines differ. Anthropic, which authored the proposal, has reportedly begun implementing support. OpenAI says it will implement the signal in the coming months. Microsoft will start honoring it on February 1, 2027, and Google aims to support it during 2027.

The scope is narrower than many publishers might hope. The token expresses a preference about training only: it does not block crawling, does not remove a site from search indexes, and does not stop AI answers from citing or quoting the page. Content already collected into existing datasets is not removed by it. A site that wants to stay out of training but keep search visibility can do so; a site that wants to stay out of AI answers needs other tools.

The providers are adding their own nuances. Microsoft is splitting training from grounding: alongside the shared token it will offer a page-level training setting and a separate AI Grounding setting that decides whether content can be used to generate answers, plus a DisallowExtended option. Old NoArchive and NoCache tags must be switched to the new controls by March 1, 2027, and sites will still appear in Bing results regardless of their preferences. Google said that when a site uses the shared token without explicitly mentioning Google-Extended, Google will treat that as disallowing Google-Extended, while sites that explicitly allow it remain eligible to appear and receive clicks through grounding in Gemini apps.

The announcement lands in a market where publishers have been improvising. The Really Simple Licensing (RSL) standard lets sites charge crawlers that visit; Cloudflare now blocks known AI crawlers by default for its customers; and litigation over training data is running in parallel on multiple fronts. Against that backdrop, a single line of robots.txt that four frontier labs agree to respect is the closest thing the open web has had to a working compromise.

The catch is enforceability. The draft is voluntary, the rollout stretches into 2027, and nothing in it stops a new entrant from ignoring the token entirely. But coordination itself is the news: until this week, opting out of AI training meant negotiating rules provider by provider. If the four signatories actually honor the two-line declaration, the marginal cost of saying no just dropped to nearly zero.