AI Training Opt-Out in robots.txt: Should Your Site Use It?

The AI training opt-out robots.txt token, __usage_ai-training, lets one rule tell Google, OpenAI, Microsoft, and Anthropic not to train AI models on your content. For most businesses, the smart move is a selective opt-out that protects premium content and keeps your AI search visibility.

Here's why I'm careful with clients on this one. In a small test shared in late October 2026, SEO consultant Cyrus Shepard pulled a site out of Google AI Overviews and Google AI Mode, and its AI impressions dropped to zero while regular organic clicks fell too. That test used a different Google control, but it shows what happens when site owners block the wrong thing.

What Is the AI Training Opt-Out Robots.txt Token?

The AI training opt-out token is a shared robots.txt rule, `User-Agent: __usage_ai-training`, that tells participating AI companies not to use your content to train generative AI models.

It comes from an IETF Internet-Draft called Machine-Readable Training Preference Signal, written by Kevin Kelley of Anthropic and published October 30, 2026. The same day, Google, OpenAI, Microsoft, and Anthropic said they will honor it. Google laid out its plan on the Google Search Central Blog.

The draft is still early. It is an individual submission at version 00, it has no formal IETF standing yet, and it expires May 3, 2027 unless it gets revised.

What the token does.

• ●It sets one training preference for every provider that supports it, so you no longer need a separate rule for each AI crawler.
• ●It works by path, so you can block /premium/ and still allow /press-kit/.
• ●It covers content fetched while the rule is live.

What the token does not do.

• ●It does not block crawling. Bots can still visit your pages.
• ●It does not pull anything back out of models that already trained on your site.
• ●It is voluntary. It states a preference, not a technical wall.

Data Table: Who Honors the Token, and When

Does Opting Out of AI Training Hurt Your AI Visibility?

Training alone, mostly no. Blocking grounding or AI search features, yes. This is where most site owners get mixed up.

Training means a company uses your content to change a model's learned parameters. Grounding means an AI pulls your live page to build and cite an answer. Search means showing up in the regular results.

The catch is Google. Google says a shared-token opt-out counts as a Google-Extended disallow when you don't name Google-Extended separately, and Google-Extended covers Gemini training and grounding in Gemini Apps and Vertex AI. Google's crawler documentation says Google-Extended does not affect inclusion in Google Search, but it doesn't name AI Overviews or AI Mode directly.

The stakes are real. Search Engine Roundtable's November 2026 Webmaster Report says AI Overviews now appear on more than 80% of Google searches. Seer Interactive's study of 53 brands and 5.47 million queries, from January 2025 to February 2026, found brands cited in an AI Overview earned about 2.1% click-through, versus 0.9% for brands that weren't cited.

Chart: Click-Through Rate When Your Brand Is Cited in AI Overviews

Being cited in a Google AI Overview more than doubles your click-through rate compared with being left out.

Average organic click-through rate on Google searches, by AI Overview presence and citation status, U.S., January 2025 to February 2026.

Source: Seer Interactive, 53 brands, 5.47 million queries, and 2.43 billion impressions, as reported by Search Engine Land, 2026.

If an AI Overview shows up, the only good seat left is inside it.

Training vs Grounding vs Search: How the AI Controls Compare

Each AI company now has more than one switch. Flip the wrong one and you can disappear from the answers your buyers read.

Comparison Chart: Which Control Does What

OpenAI already splits training from search in its crawler documentation. You can block GPTBot and still allow OAI-SearchBot, which keeps you eligible for OpenAI ChatGPT search answers.

Who should consider opting out of training.

• ●Publishers, course creators, and paywalled research sites, where content is the product.
• ●Brands with proprietary data or frameworks they sell.
• ●Anyone with a legal or licensing reason to restrict AI use.

Who should usually stay in.

• ●Local businesses, service providers, and SaaS companies that want AI to recommend them.
• ●Any site where AI answers drive leads, calls, or demos.

How to Set Up the AI Training Opt-Out Without Losing AI Search Traffic

Here's the process I use with clients. It takes about an hour for a typical small business site.

1. 1.Audit what you already have. Open your robots.txt and look for GPTBot, ClaudeBot, Google-Extended, and any NoArchive or NoCache tags.
2. 2.List your premium paths. Pick only the folders you would charge for, like courses, research, or gated guides.
3. 3.Add the shared group. Use a specific group, not a wildcard.
4. 4.Keep your current crawler rules. The draft says User-Agent: * groups never match the token, and most companies aren't live yet, so leave your GPTBot, ClaudeBot, and Google-Extended rules in place.
5. 5.Leave search and grounding bots alone. Don't block Googlebot, Bingbot, or OAI-SearchBot on pages that bring in leads.
6. 6.Fix your Bing tags before March 1, 2027. After that date, Bing stops reading NoArchive and NoCache as AI signals, and temporary grounding settings open in Bing Webmaster Tools on December 15, 2026.
7. 7.Measure before and after. Track AI citations in the Bing Webmaster Tools AI Performance report and AI referrals in GA4, using the setup from our GA4 AI traffic tracking guide.

Not sure which pages count as premium for your business? Book a strategy call and I'll map your robots.txt rules in one session. You can also see packages and pricing for my AI crawler audit, or read GEO vs SEO vs AEO for the bigger picture.

Key Takeaways

• ●The __usage_ai-training token lets one robots.txt rule set your AI training preference for Google, OpenAI, Microsoft, and Anthropic.
• ●Only Anthropic is implementing it today, so keep your existing GPTBot, ClaudeBot, and Google-Extended rules.
• ●Training, grounding, and AI search are separate controls, and blocking grounding or AI features can cost you clicks.
• ●Brands cited in Google AI Overviews earn about 2.1% click-through versus 0.9% when they aren't cited.
• ●Use path-level opt-outs for premium content and leave lead-driving pages open to AI search.

Frequently Asked Questions

What is __usage_ai-training in robots.txt?

It is a shared robots.txt policy token that tells participating AI companies not to train generative AI models on your content. You add a group that starts with User-Agent: __usage_ai-training and then use normal Disallow and Allow paths. It comes from an IETF draft published October 30, 2026.

Will blocking AI training remove my site from Google AI Overviews?

Google says Google-Extended, which the shared token maps to, does not affect inclusion in Google Search. Google AI Overviews and Google AI Mode have their own control, which is still being tested. In one small test, opting a site out of those AI features dropped its AI impressions to zero and lowered organic clicks.

When will Google, OpenAI, and Microsoft honor the AI training opt-out?

Anthropic has already started implementing the token. OpenAI says support will arrive in the coming months without a firm date. Microsoft Bing starts honoring it on February 1, 2027, and Google plans to support it in 2027. Until then, keep your existing crawler-specific rules in your robots.txt file.

Does the AI training opt-out remove my content from existing AI models?

No. The token only covers content fetched while the rule is in your robots.txt file. Anything AI companies collected before you added it stays in the models already trained on it. That makes it a forward-looking preference, so add it sooner rather than later if you plan to opt out.

Should a small business opt out of AI training?

Most small businesses should not opt out of AI training across the whole site. You want ChatGPT, Gemini, and Perplexity to know who you are and recommend you. Opt out only on paid or proprietary content, and never block the search crawlers that put your pages in AI answers.