OpenAI, Google, Microsoft and Anthropic will honor one robots.txt signal for AI training opt-out. Decide your clients' policy now

One line in robots.txt may soon tell every major AI provider whether your site can be used for training. Someone has to decide what that line says.

What happened

OpenAI, Google, Microsoft and Anthropic say they will adopt a shared signal that lets websites opt out of AI training, The Verge reports. The "policy token" was proposed at the IETF by Kevin Kelley, a technical lead for standards at Anthropic, and lives in a site's robots.txt file. Today a site has to add a separate entry for each AI provider, and for each crawler a provider runs, SiliconANGLE notes. The shared token would cover all scrapers collecting training data for generative AI models in a single entry. It would not apply to models that perform only non-generative tasks.

Anthropic has already adopted the signal, according to The Verge. OpenAI says it will implement it in the coming months, and Google plans support next year. Microsoft says it will also offer a separate AI Grounding control for when an AI fetches a page to answer a user's question, and that sites stay in Bing search results either way. Meta supports the idea but has not committed to honoring it.

My take

For most small business sites I work on, robots.txt has never been a business decision. Someone copied a template years ago. That is about to change, because there are now at least three separate choices:

1. Training: may AI companies learn from your content?
2. Grounding: may an assistant fetch your page to answer a customer's question right now?
3. Search: do you still want to rank and be found?

Those answers can differ. A service business may be fine blocking training but want assistants to read its pricing and booking pages, since a page they cannot fetch is a page they cannot point a customer to.

What I would do this month: list every client site, record what its robots.txt allows today, and get a written yes or no from the owner on each of the three. The draft is still at the IETF, so check the final syntax before rolling it out. When it lands, the change is one line per site instead of a scramble.

Site, CRM and AI visibility settings are part of the systems I set up for teams. More at romielwillautomate.dev.