Why "readable to AI" is a separate question from "looks fine to me"

A page can look perfect in your browser and be unreadable to an AI engine, because your browser runs JavaScript, applies styles, and renders the full experience — and an AI crawler often does not. What the engine sees can be a stripped-down version, or an empty shell if the real content loads only after scripts execute. So "my site looks great" tells you nothing about whether AI can read it. You have to check what the crawler sees, not what you see.

There are three gates a page passes before it can be cited: reachability, readability, and extractability. A failure at any one makes the page invisible, and each is checkable.

Check 1 — Can the AI crawlers reach your pages?

The AI crawlers reach your pages only if robots.txt lets them, so look at yoursite.com/robots.txt first and confirm it does not block the agents that make a page eligible to be cited: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot (Perplexity), and Googlebot, which is what crawls for Google Search and therefore for AI Overviews and AI Mode as well. A common, costly mistake: a site blocks ClaudeBot to keep its content out of AI training and accidentally blocks the search and fetch crawlers too, making itself uncitable. Google-Extended belongs in a different category and is covered in the table below, because it is not a crawler at all. Also check that a security layer (a firewall or bot-blocking rule) isn't silently blocking these agents; that produces zero visibility with no visible error. Robots.txt is a statement of permission rather than proof of access, so treat it as the first half of this check: which agent does what is set out in the table below, and whether each one actually arrived is settled in your server logs.

Which AI crawler user agents should you allow, and what does each one control?

Each engine runs more than one user agent and they control different things, so allowing the AI crawlers is several decisions rather than one. OpenAI and Anthropic both split theirs three ways: one agent collects training data, one builds the search index that makes a page eligible to be cited, and one fires only when a person pastes a URL and asks for it to be read. The settings are independent, so blocking the training agent leaves the search agent free to index you, and blocking the search agent is the one that takes you out of the answers (OpenAI’s overview of its crawlers, read 2 October 2026).

AI crawler user agents, what each one does, and what blocking it actually costs

User agent	Run by	What it does	What blocking it costs you
GPTBot	OpenAI	Collects training data for the foundation models	Nothing in ChatGPT’s search answers
OAI-SearchBot	OpenAI	Builds the search index ChatGPT retrieves and cites from	Eligibility to be cited in ChatGPT
ChatGPT-User	OpenAI	Fetches one page when a person asks ChatGPT to read that URL	The ability for a person to show ChatGPT your page
ClaudeBot	Anthropic	Collects training data	Nothing in Claude’s search answers
Claude-SearchBot	Anthropic	Indexes pages for Claude’s search answers	Eligibility to be cited in Claude
Claude-User	Anthropic	Fetches a page a person asked Claude to read	The same user-initiated read
PerplexityBot	Perplexity	Indexes pages for Perplexity’s answers	Eligibility to be cited in Perplexity
Perplexity-User	Perplexity	Fetches a page on a user’s request	The user-initiated read
Googlebot	Google	Crawls for Google Search, which is also what makes a page eligible for AI Overviews and AI Mode	Everything in Google, the AI surfaces included
Google-Extended	Google	A robots.txt control rather than a crawler: it governs whether already-crawled content may be used to train and ground Gemini	Nothing in Google Search or its AI surfaces

The last row is the one that trips people up, in both directions. Google-Extended has no HTTP user agent of its own and will never appear in a server log, because the fetching is done by Google’s existing crawlers and the token only governs what may be done with the content afterwards (Google’s documentation on its common crawlers, read 2 October 2026). Disallowing it does not reduce crawling, change indexing or affect ranking, and it does not keep a page out of AI Overviews or AI Mode, which are features inside Search rather than separate products. Eligibility there follows from being indexed and snippet-eligible, and the controls that limit it are nosnippet, data-nosnippet, max-snippet and noindex. So a robots.txt that disallows Google-Extended and nothing else has blocked no AI crawler, and a site hoping to be cited has gained nothing by allowing it.

How do you confirm from your server logs that the AI crawlers are reaching you?

Server logs are the only record that shows whether the AI crawlers actually arrived, because robots.txt states a permission and a log states a fact. Search the access log for the agent strings in the table above and read three things off every hit: which agent, which URLs it asked for, and the status code it got back. A 200 means the page was served. A 403 or a 429 means something in front of the site refused it, which is the failure a robots.txt check cannot see and is usually a firewall or bot-management rule rather than anything you wrote.

The more useful finding is an agent that is absent altogether. A search agent that has never requested a page cannot have indexed one, and no amount of on-page work changes that until the request succeeds. Give the window enough length to mean something, because these agents crawl unevenly: a week of logs is a weak signal on a small site and a month is a better one.

Accidental blocking is also more common than most site owners assume. Ahrefs measured robots.txt across roughly 140 million websites and found GPTBot the most blocked AI bot at 5.89% of sites (Ahrefs, AI bot block rates across ~140 million websites, read 2 October 2026). Block rates diverge sharply by sample, with large publishers far above the long tail, so the number that decides anything for you is the one in your own log rather than any headline figure.

Check 2 — Does your content render without JavaScript?

Content that renders only with JavaScript is the most common silent failure. To test it: open one of your key pages, view the page source (right-click → "View Page Source", which shows the raw HTML before scripts run), and search for a sentence you know is on the page. If the text is there, the engine can likely read it. If the source is nearly empty and the content only appears in the rendered view, your page is JavaScript-dependent — and an AI crawler may see an empty shell. You can also disable JavaScript in your browser and reload: if the content vanishes, so does it for the engine.

This matters more for AI than it does for Google, because most AI crawlers do not run JavaScript at all. A joint study by Vercel and MERJ covering more than 500 million GPTBot fetches found no evidence of JavaScript execution, and the same held for Anthropic's ClaudeBot, Perplexity's PerplexityBot, Meta's agent and ByteDance's crawler. The detail that misleads people is that these crawlers do fetch the files: ChatGPT's agents requested JavaScript on 11.50% of requests and Claude's on 23.84%, without ever executing them (Vercel and MERJ, “The rise of the AI crawler”, read 2 October 2026). So a log full of JavaScript requests is not evidence that a page was rendered. Googlebot is the exception among the major crawlers in rendering JavaScript, which is how a client-rendered page can appear in Google Search and be absent from every AI answer at the same time.

Check 3 — Can the engine extract a clean answer?

Reachable and readable get you considered; extractable gets you cited. Engines prefer content they can lift as a self-contained unit: a direct answer near the top of a section, under a clear heading, backed by specific facts. Check whether your important pages open with the answer or bury it under paragraphs of preamble, whether headings are real questions or vague labels, and whether claims are specific and verifiable rather than vague superlatives. A page that makes the engine hunt for the answer is far less likely to be cited than one that hands it over cleanly. (The AI citation checklist breaks the extractable structure down signal by signal.)

Check 4 — Ask the engines directly

The most direct test: ask each engine to read your page. In ChatGPT, Perplexity, or Claude, paste your URL and ask "what does this page say about [topic]?" If the engine can summarize it accurately, it can read and extract it. If it returns nothing, a generic guess, or an error, that's a reachability or rendering problem worth running down. Do this across engines — because they read differently, a page readable to one may fail on another.

What to fix if AI can't read your site

Each fix maps to the check that failed, so work from whichever of the four told you the site is unreadable. A reachability failure means correcting robots.txt or a firewall rule. A readability failure means server-side or static rendering so content is in the HTML, not script-dependent. An extractability failure means restructuring pages — direct answers up top, question-based headings, specific facts. These are the technical floor of AI visibility; clearing them doesn't guarantee citation, but failing them guarantees its absence. (For how the engines weigh what they can read, see how AI engines choose sources.)

Frequently asked questions

Related guides

• AI citation checklist: 25 signals that matter — the structural signals behind extractability.
• How AI engines choose which sources to cite — what happens after a page is readable.
• Why ChatGPT ignores your website — the deeper diagnosis when a page is readable but still skipped.
• The 7 SIGNALS dimensions — the full model behind AI citation.