Should you block AI crawlers like GPTBot?
Blocking GPTBot stops OpenAI training on your content. It does not stop ChatGPT citing you, because a different bot does that job. If you sell something and want to be named in AI answers, block the training crawlers if you like and leave the retrieval crawlers open. If you sell page views, the arithmetic is different and blocking is defensible. The one thing nobody should do is block everything with an AI-sounding user agent and then wonder why the engines stopped mentioning them.
Should you block AI crawlers, and what happens if you do?
Block AI crawlers selectively, not as a category, because the label covers two jobs that pull in opposite directions. Training crawlers copy your content into a dataset that improves a model, and you get nothing back that you can point at. Retrieval crawlers fetch your page so the engine can quote it and link to you in an answer, which is the entire mechanism by which a business gets recommended by ChatGPT or Perplexity. Blocking the first costs you nothing measurable. Blocking the second removes you from the answer.
The decision therefore turns on what your site is for. A publisher whose revenue is advertising against page views loses money on every crawl that produces no visit. A business whose website exists to get it shortlisted loses far more by being absent from answers than it spends on bandwidth. Same file, same directives, opposite conclusions, and most of the advice online fails to separate the two cases.
The infrastructure has already taken a side for a large part of the web. On 1 July 2025 Cloudflare began blocking AI crawlers by default for new domains and opened a Pay Per Crawl beta letting publishers charge for access, a change that applies across a network carrying roughly 20% of web traffic.
Which AI crawlers actually decide whether you get cited?
The AI crawlers that decide citation are the retrieval and user-triggered fetchers, not the training bots that get all the attention. OpenAI runs three, and the distinction between them is the single most useful thing to understand before editing a robots.txt file. GPTBot gathers training data. OAI-SearchBot builds the index behind ChatGPT search. ChatGPT-User fetches a page live when a person's question requires it.
OpenAI's own crawler documentation is explicit that sites which disallow OAI-SearchBot will not be shown in ChatGPT search results, and equally explicit that GPTBot is about model training. Blocking GPTBot while allowing OAI-SearchBot is a coherent position: no training, full citation eligibility.
| Crawler | Operator | What it does | Blocking it costs you |
|---|---|---|---|
| GPTBot | OpenAI | Collects training data for future models | Nothing you can measure in citations |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search | Your place in ChatGPT search answers |
| ChatGPT-User | OpenAI | Fetches a page in response to a user's request | Live lookups of your pages during a chat |
| ClaudeBot | Anthropic | Crawls content for Claude | Claude's ability to reach your pages |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers | Citations in Perplexity, which links heavily |
| Google-Extended | Governs use of your content for Gemini training and grounding | Gemini uses, but not Search or AI Overviews | |
| Googlebot | Crawls for Search, and feeds AI Overviews | Search and AI Overviews together |
Google-Extended is the one people most often get wrong in the other direction. It is a robots.txt token rather than a crawler with its own user agent, it governs whether content can be used for Gemini training and grounding, and Google's crawler documentation states that it does not affect inclusion in Google Search and is not used as a ranking signal.
What does AI crawling actually cost you?
AI crawling costs bandwidth and origin load, and for content businesses it costs the visit that used to arrive with the crawl. The metric that made the imbalance legible is the crawl-to-refer ratio: how many pages a platform fetches for every visitor it sends back. Traditional search crawls a page and eventually delivers clicks. Answer engines crawl far more and, because the answer resolves in the chat window, return far fewer.
Cloudflare published the numbers when it launched the metric on Radar in July 2025. In "The crawl before the fall of referrals", Anthropic's crawl-to-refer ratio was roughly 70,900:1 for the week of 19 to 26 June 2025, against 887:1 for OpenAI and 118:1 for Perplexity over the comparable default window.
Most of that traffic is not the kind that can cite you. Cloudflare's analysis of crawler purpose, "From Googlebot to GPTBot", found that by July 2025 about 79% of AI crawler requests were for training, 17% for search, and 3.2% triggered by a user action, compared with 72% training and 26% search a year earlier.
Those two measurements point the same way. The overwhelming majority of the load comes from crawlers that will never send a reader back, and a thin slice comes from the ones that decide whether you appear in an answer. Blocking by category throws away the slice that pays in order to stop the bulk that does not, while blocking by purpose keeps it.
How do you allow citation crawlers and block training crawlers?
Allowing citation crawlers while blocking training crawlers takes a handful of robots.txt directives, written per user agent rather than as a blanket rule. Order the file so that the training bots are named and disallowed, the retrieval bots are named and allowed, and Googlebot is left entirely alone unless you want to leave Google Search as well.
User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Googlebot Allow: /
Two cautions before you copy that. Robots.txt is a convention that compliant crawlers honour voluntarily, so it is a signpost rather than a lock, and anything you genuinely need to prevent belongs behind rate limiting, bot management or authentication at the edge. And a directive only helps if the rest of the page is reachable: a retrieval crawler that is welcomed by robots.txt and then served an empty shell that fills in through JavaScript has still learned nothing about your business.
Check the result from outside rather than trusting the file. Fetch your own page with each user agent, confirm the HTML that comes back contains the text you expect, and repeat it after any CDN or bot-management change, because a security rule added months later is the most common way sites lose citation eligibility without noticing. Our guide to checking whether AI can read your website walks through the tests.
Does llms.txt help AI crawlers find your content?
An llms.txt file is close to inert today, and no robots.txt decision should be made on the assumption that it compensates. The idea is reasonable: a markdown file at the root that lists your important pages in a form a model can digest. The problem is adoption on the other side, since no major engine has committed to reading it in production.
The measurement is unambiguous. Ahrefs analysed 137,000 sites and found that 97% of valid llms.txt files received no requests at all during May 2026, with AI retrieval bots accounting for roughly 1.1% of the requests that did arrive. Most of the traffic came from SEO tools and validators checking whether the file existed.
Keep yours if you already publish one, as we do. It costs nothing, it does no harm, and if an engine starts honouring the convention you are ready. Treat it as a bet with a small stake rather than as part of your access strategy, and spend the attention on the retrieval crawlers instead, which are demonstrably fetching pages and demonstrably deciding who gets named.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
Blocking GPTBot does not remove you from ChatGPT's answers, because GPTBot collects training data rather than powering search results. The bot that decides whether ChatGPT can retrieve and cite your page when someone asks a question today is OAI-SearchBot, and OpenAI's crawler documentation states that sites which disallow OAI-SearchBot will not be shown in ChatGPT search results. Blocking the wrong one of the two costs you nothing you wanted to keep; blocking the other one costs you the channel.
Is robots.txt legally binding on AI companies?
No. Robots.txt is a voluntary convention, not an access control, and it works only for crawlers that choose to honour it. Enforcement happens at the server or the CDN, through rate limiting, bot management or authentication. If your reason for blocking is legal or commercial rather than technical, treat robots.txt as a statement of intent and put the actual enforcement somewhere it can be enforced.
Does blocking Google-Extended affect my Google rankings or AI Overviews?
No. Google-Extended controls whether your content can be used to train and ground Gemini models, and Google's documentation states that it does not affect a site's inclusion in Google Search and is not used as a ranking signal. AI Overviews are served from the standard Google index and are fetched by Googlebot, so a page blocked to Google-Extended can still be crawled, indexed and cited in an AI Overview.
Should publishers block AI crawlers if AI sends no traffic?
Publishers whose revenue depends on page views have a real case for blocking training crawlers, because the exchange is lopsided. According to Cloudflare's Radar data, published when it launched the crawl-to-refer metric in July 2025, Anthropic made roughly 70,900 page requests for every referral it sent back in the week of 19 to 26 June 2025. Businesses that sell something other than page views usually have the opposite case, because being named in an answer is worth more than the bandwidth the crawl consumes.
Does an llms.txt file help AI crawlers find my content?
The evidence says it does almost nothing. Ahrefs analysed 137,000 sites and found that 97% of valid llms.txt files received no requests at all during May 2026, and that AI retrieval bots accounted for about 1.1% of the requests that did arrive. Publishing one is cheap and harmless, but it is not a substitute for letting the retrieval crawlers reach your actual pages.