AI training controls in robots.txt: what changes, and what we set.

On 30 October 2026 Google said it plans to support a shared AI training control in robots.txt: one rule that tells every participating AI developer not to train on your site, instead of one rule per company. It doesn't change who can cite you, it doesn't touch AI Overviews, and the line itself isn't published yet. This page covers what the control does, why one robots.txt line can't settle training, search and live fetching at once, and the file we run on this domain, quoted as it stands.

What is the shared AI training control, and what is it not?

It is a policy token for robots.txt that states a site's AI training preference to every provider that adopts it. Google's Search Central post says the definition of AI training comes from the IETF AIPREF working group, in the draft draft-kelley-aipref-training-preference-signal, and that the rest of the AIPREF vocabulary is still under discussion.

Google says that if a site uses the token to opt out of training and doesn't name Google-Extended, Google will treat it as if Google-Extended were disallowed. Company-specific controls keep working alongside it. Google aims to support it in 2027, with other providers, and calls it a proposal it will keep refining.

Going by the post itself, it isn't live yet: Google gives a target year, not a date. The post doesn't print the token or a sample line either, so we won't. When Google documents it, the line goes in the file; until then a guessed directive is a line no crawler reads. And it is about training only. Whether a page can be quoted in an answer is a different setting, covered next.

Why can't one robots.txt line cover training, search and live fetching?

Because each AI company runs separate agents for separate jobs, and a rule only reaches the agent it names. Blocking a company's training crawler leaves its search crawler alone, and the reverse. The table lists what each vendor documents.

Two details in those pages change how you write the file. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person started the request. Perplexity says Perplexity-User generally ignores robots.txt for the same reason. So the live-fetch column is the one robots.txt controls least.

Google is the odd one out. Google-Extended has no user agent of its own; Google crawls with its usual agents and reads the token as a permission. Google's AI features guide says robots.txt rules for Googlebot are the control for Search, AI features included, and that limiting what Search shows from a page is done with nosnippet, data-nosnippet, max-snippet or noindex. Its Oct 30 post adds that sites allowing Google-Extended are eligible to appear in Gemini Apps answers through grounding. Disallowing it therefore costs you that.

What does our robots.txt say today?

Everything is allowed, on purpose. This is the top of the live file, without its comments:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

User-agent: * with Allow: / lets every crawler fetch every public path. There are no Disallow rules. The two private paths on the server refuse requests on their own, so the file doesn't need to list them.

Content-Signal is Cloudflare's format, from its Content Signals Policy. It declares three preferences: search for building a search index, ai-input for feeding content into a model at answer time, such as grounding, and ai-train for training or fine-tuning. Cloudflare says plainly that these express preferences and are not technical countermeasures. Ours says yes to all three, to match the Allow rules.

Below that, the file names each AI agent in its own group with Allow: /: GPTBot, ChatGPT-User and OAI-SearchBot; ClaudeBot, Claude-SearchBot, Claude-User and the older anthropic-ai token; PerplexityBot and Perplexity-User; CCBot, Applebot, Applebot-Extended, Bytespider, cohere-ai and Google-Extended. The wildcard already allows them, so the named groups change nothing today. They exist so that a later block or crawl delay can target one agent without touching the rest, and because "allowed by omission" reads like an oversight to anyone who checks.

While writing this we found our own comment on Google-Extended was wrong. It described the token as covering AI Overviews grounding. Google's crawler reference says Google-Extended doesn't affect inclusion in Search, so we corrected the comment the same day. Claude-SearchBot, Claude-User and Perplexity-User were added then too, since their vendors document them and the file didn't list them.

Why do we allow AI search and live-fetch agents?

Because being cited is the point. This firm sells technical GEO, and a page that an assistant's search crawler can't fetch can't be quoted in that assistant's answers. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot takes a site out of those answers. OpenAI says so directly about OAI-SearchBot.

The live fetchers matter more than their reputation suggests. In our twenty-week benchmark's crawler data, ChatGPT-User made 18,871 requests across 14 properties in seven days in August 2026, against 433 for Google-Extended. A live fetch happens because a person just asked a question, so it is the closest thing in a crawler log to a reader.

Training is a separate choice. We allow it because this site publishes to be read and quoted, and we have no content we'd want kept out of a model. A site with a different business, paid research for example, could reasonably decide the other way.

Do robots.txt rules stop impersonators?

No. Robots.txt is a request that a well-behaved crawler reads and honors. A scanner that borrows a crawler's name doesn't read it, and on this domain most traffic carrying an AI-crawler name is exactly that.

We split the requests by path. Of 1,180 requests carrying the Applebot string, 1,171 were credential probes for files no site should publish, such as /.ssh/id_rsa, and 9 asked for a real page; all 9 were served. Across the seven days to 17 August 2026, 1,499 of the 1,972 requests presenting an AI-agent string asked for a path this site has never published. Both measurements are written up with their method in the benchmark and our GEO essay.

The real control sits at the edge. A CDN such as Cloudflare classifies verified bots by more than the user-agent string, so a firewall rule can serve the real crawler and refuse the impostor using its name. In the August measurement, every request Cloudflare verified as a real AI crawler got a 200 or a 301, and every refusal on a page that exists went to an unverified impostor. Write robots.txt for the crawlers that ask, and enforce at the edge for the ones that don't.

Which directives match which goal?

Three common positions, and the rules that express each one with the controls that exist today. Add the shared training token once Google publishes its syntax.

Whatever you choose, put a firewall rule behind it if the goal involves blocking anything, since the file alone won't stop a scraper that ignores it.

Questions about AI training controls

Does blocking Google-Extended remove my site from AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews and AI Mode are part of Search, so they follow Googlebot. To limit what Search shows from a page, Google points to nosnippet, data-nosnippet, max-snippet and noindex.

What is the robots.txt line for the shared AI training control?

Google's 30 October 2026 announcement does not publish the token or its syntax. It comes from an IETF AIPREF draft that is still being revised, and Google says it aims to support it in 2027. Until Google documents the line, any syntax you see quoted is a guess.

Is robots.txt enough to stop my content being used for AI training?

It is a request that the named, well-behaved crawlers honor. It does nothing about a scraper that ignores it or borrows a crawler's name. Enforcement happens at the edge, where a CDN can tell a verified crawler from an impostor.

Should a small business block AI training?

Only if training is the thing you object to. Blocking the training agents costs little in visibility, but blocking the search and fetch agents takes you out of the answers those assistants give. Decide per agent, not per company.

Sources, and reusing this page

Claims about what a control or an agent does link to the vendor's own page where they are made: Google's Search Central post of 30 October 2026, its AI features guide and its common crawlers reference; OpenAI's bots reference; Anthropic's crawler article; Perplexity's bot guide; and Cloudflare's Content Signals Policy. We read all of them on 2 November 2026. The robots.txt excerpt is copied from the live file on that date. Crawler counts come from Cloudflare edge analytics and are published, with their windows and method, in the articles linked beside them; no new figures were measured for this page.

You may republish or adapt the decision table with attribution and a link to https://concept211.com/articles/ai-training-controls-robots-txt/. Each heading and each table row carries a stable id.

Photo by panumas nikhomkhai on Pexels