How to Control AI Crawlers on Your Site: robots.txt, Google-Extended, and What Each Toggle Really Does

"Should we block the AI bots?" has become a standing agenda item, usually asked in a tone somewhere between principle and panic. It is a reasonable question with a genuinely complicated answer, and most of the confusion comes from one misunderstanding.

The takeaway up front: there is no single "AI crawler" switch, because different bots do different jobs for the same company — and the controls are voluntary. Blocking a token that governs model training does not remove you from AI answers in search. Blocking the crawler that feeds search removes you from search entirely. Getting the distinction right is most of the work; deciding your policy is the easy part once you can see what each lever does.

This is an explainer of the mechanism, not coverage of any particular announcement. Every control described here is one the vendor publishes in its own documentation, and the sensible habit is to check that documentation rather than a screenshot, because these tokens change.

First: the three jobs crawlers do

Group the traffic by purpose, not by company. Most AI-related crawling falls into three buckets:

1. Search indexing. The classic job: fetch pages so they can be indexed and ranked. Google states that AI Overviews draw on its search index, which means the search crawler is also what makes a page eligible to appear in those answers.
2. Model training. Fetching content to build or improve a generative model. This is the bucket most publishers actually have feelings about.
3. User-initiated and grounding retrieval. An assistant fetching a page right now because a user asked something, or to ground an answer with live citations.

The reason blanket blocking backfires is that these three run under different user agents, and some vendors deliberately separate them so publishers can make different choices. If you want the citation traffic but not the training use, that distinction is your friend.

The controls that actually exist

robots.txt — the primary lever, and its real status

The Robots Exclusion Protocol is standardised as RFC 9309, published through the IETF. The standard describes a voluntary protocol: compliant crawlers read /robots.txt and honour it. Nothing in it enforces anything. That is the single most important property to internalise — robots.txt is a request, not a fence.

The syntax is per-user-agent, matching on the product token:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

A few mechanical points people get wrong:

• A crawler obeys the most specific matching group only. Write a GPTBot group and that bot ignores your User-agent: * rules entirely, so anything you wanted applied to it must be repeated in its own group.
• Disallow blocks crawling, not indexing of the URL itself. To keep a page out of an index, use a noindex robots meta tag or X-Robots-Tag header — which requires the crawler to be allowed to fetch the page and see it.
• robots.txt is per host and per protocol. https://example.com/robots.txt does not govern https://blog.example.com/.

The vendor tokens worth knowing

Each of these is documented by the vendor that operates it; the descriptions below reflect that published documentation, and it is worth re-reading the source before you deploy, because scope has been revised before.

• Googlebot — crawls for Google Search. Blocking it removes you from Google Search. Because AI Overviews are served from the search index, blocking Googlebot to avoid AI answers also costs you the organic listings.
• Google-Extended — a standalone token that does not crawl on its own. Google documents it as controlling whether content already fetched is used to help improve its Gemini apps and Vertex AI generative APIs, and states explicitly that Google-Extended has no effect on inclusion or ranking in Google Search. It is the closest thing to a training-only opt-out in Google's stack.
• GPTBot — OpenAI's documented crawler for gathering content that may be used in model training.
• OAI-SearchBot — OpenAI's documented crawler for surfacing and linking sites in ChatGPT's search features. Distinct from GPTBot, and this is the one to keep allowed if you want to be citable.
• ChatGPT-User — documented as user-initiated fetching, when a person asks the assistant to visit a page.
• Applebot and Applebot-Extended — Apple documents Applebot for Siri and Spotlight, and Applebot-Extended as the opt-out for using content in its generative models.
• Anthropic's published crawler tokens, which likewise separate training-oriented crawling from user-initiated fetches.
• CCBot — Common Crawl, a nonprofit web archive whose corpus has been widely used as training data by many parties. Blocking it affects an archive, not one model.

Treat that list as a starting point. New tokens appear, existing ones get re-scoped, and the only reliable version is the vendor's own documentation page.

Snippet controls — the underused middle ground

Between "fully open" and "blocked" sit Google's snippet directives, which the company documents as applying to AI Overviews as well as ordinary results: nosnippet, max-snippet:[n], and the data-nosnippet HTML attribute for marking specific passages.

This is a genuinely different lever. It does not remove you from the index or the answer; it limits how much of your text can be shown. For publishers whose objection is "my article is being reproduced," not "I don't want to be found," it is often a better fit than a blanket disallow — with the honest trade-off that a shorter or absent snippet can reduce click-through.

Network-level blocking

robots.txt asks. WAF rules, rate limits, and bot-management products enforce, by refusing the request. Infrastructure vendors including Cloudflare have shipped managed AI-crawler blocking and, per their own announcements, options making it the default for new domains, alongside experiments in charging crawlers for access.

Enforcement is the only option with teeth, and it carries costs: aggressive bot management can catch crawlers you wanted and produce measurement gaps — a pattern covered in our piece on how anti-bot systems reshape what your tools can measure.

llms.txt — a proposal, not a standard

llms.txt is a community proposal for a markdown file describing a site's content for language models. Worth knowing the name, because clients ask. The accurate status: it has some adoption among tooling vendors, no major search engine has documented using it, and Google representatives have publicly said it is not used by their systems. The IETF's AI preferences work is where a real standardised opt-out signal would come from. Publishing an llms.txt costs almost nothing and currently does almost nothing — treat it as a bet, not a control.

How to decide

Skip the ideology and answer four questions about your own site.

1. Where does your traffic and revenue come from? If organic search matters, Googlebot and the equivalents stay allowed. That is not really a debate.
2. Do you want to be cited in AI answers? If yes, keep the search and retrieval bots allowed — the OAI-SearchBot-style tokens, not just the training ones. If your business is publishing and citations bring readers, blocking retrieval bots is self-harm.
3. Is your objection specifically to training use? Then the training tokens are your lever: Google-Extended, GPTBot, Applebot-Extended, CCBot, and equivalents. This is the most common publisher position and it is cleanly expressible.
4. Do you have content that must not be reproduced at length? Use the snippet controls, and consider genuine enforcement at the edge for the specific paths that matter — usually a paywalled or licensed archive, not the whole site.

A common landing point for a content business: allow search and retrieval crawlers, disallow training-only tokens, apply data-nosnippet to premium passages, and leave enforcement for the small number of paths where it genuinely matters.

What to check after you change anything

• Test the file. A syntax error in one group can silently change behaviour for everything.
• Confirm you did not break search. Watch coverage and crawl stats in your search console of choice for a few weeks; the risk of an over-broad rule is losing indexing, and that is a slow, quiet failure.
• Log what you did and when. When traffic moves three months later, "did we change robots.txt?" should be answerable in seconds.
• Verify the bots you see are the bots they claim to be. Major crawlers publish IP ranges or support reverse-DNS verification; a user agent string alone proves nothing.

For wider context, our map of the modern search landscape covers where AI answers sit in the pipeline, and how AI search is changing SEO covers the strategic response once your crawler policy is set.

FAQ

Does blocking Google-Extended hurt my rankings?

Per Google's own documentation, no: Google-Extended does not affect inclusion or ranking in Google Search. It governs whether content is used to improve Gemini apps and Vertex AI generative APIs. Blocking Googlebot is an entirely different action and would remove you from Search.

Can I stop my content appearing in AI Overviews without leaving Google Search?

Not by blocking a separate token, because AI Overviews draw on the search index. The available middle ground is Google's snippet controls — nosnippet, max-snippet, and data-nosnippet — which Google documents as applying to AI Overviews and which limit how much text can be shown, at the cost of reduced snippets in ordinary results too.

Do AI crawlers actually obey robots.txt?

The protocol is voluntary by design, as RFC 9309 makes explicit. Major documented crawlers state that they honour it, and infrastructure vendors and trade press have reported cases where crawling did not match published rules. If you need compliance you can rely on, enforcement has to happen at the network layer, not in a text file.

Is publishing an llms.txt file worth doing?

It costs little and currently delivers little. No major search engine has documented using it, and Google representatives have said publicly that their systems do not. Treat it as an optional, low-cost hedge and do not let it substitute for the controls that vendors actually document.

Crawler tokens, snippet directives, and AI-search behaviour all move faster than any evergreen guide. Track crawler and AI-search changes as they are announced, with every source shown, on Moz News.