Why does a 200 response not mean an AI crawler read your page?

A 200 response describes the delivery and says nothing about the contents, so a crawler can be handed a challenge screen, an empty shell or a placeholder and record the fetch as a success.

Zhang's study put a size on how often that happens. Fetching 10,307 URLs from 1,507 sites across three days with a polite static client and a rendered browser client, and labelling what the successful responses actually contained, 20.9% of the pages a static client received with status 200 did not contain the page's main content, with a reviewed held-out sample putting the figure at 20.0% (arXiv 2611.00884, 31 October 2026). Roughly one static fetch in five lands on something unusable.

The second finding in the same study is the one that makes testing harder than it looks. 13.1% of sites whose random-path probe was answered returned a 200 for a path that does not exist. On those sites a 200 carries no information at all, because the server returns one whatever you ask for.

Why this matters more for answer engines than for Google is a question of what each one does next. Google has run a rendering pipeline for years and has infrastructure for retrying. Most AI crawlers take the raw HTML in the initial response and work with that, so whatever arrives in the first payload is the entire page as far as the engine is concerned.

Worth separating from a ranking problem, because the fix is completely different. A page competing badly is a content question with a content answer. A page handing over nothing is an access question that sits earlier in the chain, and no amount of writing changes it.

What kinds of page serve a 200 with no content?

Five kinds of page commonly serve a 200 with nothing an engine can use, and all five look fine when you open them in a browser.

What arrives	Why it has a 200	Who usually caused it
Bot challenge or interstitial	The challenge is itself a successful page	Bot management at the CDN or firewall
Client-rendered shell	The HTML is valid and the content arrives later by script	A single page application or client-side framework
Consent or region wall	The wall is served in place of the document	Privacy tooling, geo rules
Soft 404 or placeholder	A missing page answered with a friendly page	Content management defaults, old redirects
Partial content	Headers and navigation render, the body does not	A failed component, a timed-out include

Two of those five are worth more attention than the others because they are the ones a marketing team cannot see and cannot fix alone.

Bot management is the most common and the most invisible. A firewall rule that issues a JavaScript challenge to unfamiliar clients will issue it to crawlers that cannot run JavaScript, and those crawlers then read your homepage as empty while every human visitor sees it normally. The rule frequently differs per path, so a site can be readable at the root and blind on the pages that matter.

Client-side rendering is the other, and it has its own guide here, because the symptom and the test are different: our page on whether AI crawlers read JavaScript covers which crawlers render and which do not, and what a client-rendered page looks like to ChatGPT.

How do you check your own site in an afternoon?

Check your own site with three cheap tests, which the study above validated against a human-reviewed sample, and then confirm the worst offenders by hand.

The three tests are text length, a probe against a nonsense URL, and a check for redirects to the homepage. Combined and tuned on held-out sites, those three recovered 86% of unusable responses at a false-alarm rate near 5%, and a text classifier added only five to seven points of recall at more than twice the false-alarm rate (arXiv 2611.00884). Three crude heuristics catch most of the problem, which is unusual and useful.

In practice, run each check this way. For text length, request the page with a crawler user agent, take the raw HTML without executing scripts, strip the tags and count the words; compare that against what a browser shows. A page that renders twelve hundred words for a person and yields forty in raw HTML is the shell case.

For the probe, request a URL on your own domain that certainly does not exist, something like a long random string, and record the status and the body. A 200 means your server's status codes carry no information, and it means every monitoring tool you own is reporting success it has not verified.

For the redirect check, follow redirects and see where you land. A deep page that quietly becomes the homepage has lost its content as surely as one that returns nothing, and the engine indexes whatever it arrives at.

Then do the part no heuristic replaces: ask somebody with server or CDN access for the response codes and bot-management actions for the real AI crawler user agents over the last month. Our broader checklist for this sits in how to check if AI can read your website, and this page is the part that checklist cannot infer from the outside.

Does robots.txt tell you whether AI crawlers can read your site?

Robots.txt tells you almost nothing about whether AI crawlers can read your site, because the decisive controls now sit at your network edge and because one important crawler is documented as not bound by the file.

OpenAI's bot documentation is explicit on the last point. Of ChatGPT-User, the agent behind user-initiated visits, it states: "Because these actions are initiated by a user, robots.txt rules may not apply". For search inclusion the relevant agent is a different one: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers", with robots.txt changes taking around 24 hours to apply. The same page states that "Each setting is independent of the others", so blocking the training crawler does not change search visibility and blocking the search crawler does not block training collection (read 9 November 2026).

The second reason is that the decision may not be in your repository at all. Cloudflare announced on 1 July 2025 that it was "changing the default to block AI crawlers unless they pay creators for their content". A site can publish a permissive robots.txt and still have the fetch refused or challenged one layer above the origin, by a setting nobody on the marketing side knows exists.

Four places decide this between them, and robots.txt is only the first of them. Robots.txt carries the stated preference. The CDN or firewall decides what actually happens to each crawler. Server logs show whether the crawlers arrive and what they receive. The page itself decides whether the content is in the first response at all.

If the question you are actually asking is whether to allow these crawlers at all, that is a separate decision with its own tradeoffs, and our guide on whether you should block AI crawlers like GPTBot works through it.

Does an empty 200 explain why you are never cited?

An empty 200 explains it completely in a minority of cases, and it is the right thing to rule out first because it is cheap to test and absolute in its effect.

Google's requirements for appearing in its AI features are short. Its documentation states that a page must be "indexed and eligible to be shown in Google Search with a snippet", that "There are no additional technical requirements", that "You don't need to create new machine readable files, AI text files, or markup to appear in these features" and that "There's also no special schema.org structured data that you need to add" (read 9 November 2026). A page that hands over no content fails the first clause and nothing else on the list can compensate.

For ChatGPT the dependency is sharper still. Mahe Chen and Nick Koudas, studying 158,099 generative engine answers carrying 468,037 citations, state that every page ChatGPT cites is one its own search retrieved (arXiv 2610.15033, submitted 10 October 2026). A document that cannot be read cannot enter that set.

Honesty about the bound matters as much as the point. Roughly one static fetch in five landed on unusable content in Zhang's sample, so four in five sites reading this do not have this problem, and for them the explanation of a missing citation lies in retrieval, in competition or in what the model already knows. Chen and Koudas also found that retrieval explains about 32% of citation variation against 2.8% for page properties, so clearing an access fault is a precondition rather than a strategy.

Use it as a gate. Confirm the engines receive your content, then spend the rest of the budget on the things that decide between readable pages. Doing it the other way round is how a content programme runs for two quarters against a firewall rule.

How does an empty 200 break your AI visibility reporting?

An empty 200 makes a monitoring tool confidently wrong rather than silent, which is the most expensive kind of wrong, and the study quantifies the damage.

Treating a 200 as evidence of content overstated the share of claim and source pairs checkable against live content by 12.2 percentage points, and inflated the apparent rate of silent citation drift from 5.8% to 21.9% in Zhang's analysis of 4,795 URLs cited by expert-verified claims (arXiv 2611.00884, 31 October 2026). A measurement pipeline that does not distinguish an empty success from a real page reports problems that are not there and misses ones that are.

The same fault applies to any readiness or visibility score computed by fetching your pages. A tool that scores an app shell produces a confident low score for a page that is fine, or a confident high score for a page it never actually saw. Neither failure announces itself: the report arrives looking normal.

Our own position on that is a product rule rather than a preference. The SIGNALS scoring model abstains rather than inventing a result when there is not enough content to ground one, which is why, as our agency comparison puts it, a three-page site gets told to publish something real before it gets a fix list. An empty 200 is the same situation arriving down a different path, and the correct output for it is "no content was returned" rather than a number.

So add one question to any vendor conversation about measurement. Ask what the tool does when a fetch returns 200 with no main content, and whether that case appears in the report at all. A vendor who has not considered it is scoring roughly one page in five on something other than your page.

What should you fix first, and in what order?

Fix the access problems first, in the order below, because each step is wasted effort if the one above it is still broken.

Start with bot management. Exempt the AI crawlers you want from challenges, keeping the rest of your screening in place, and verify the exemption from outside rather than trusting the configuration screen. Some firewalls verify crawlers by address rather than by the user agent string, so a request that merely claims to be a crawler can be treated differently from the real one, which makes an outside test with a spoofed agent an incomplete check.

Second, get the main content into the first response. Server rendering, static generation or pre-rendering all work; which one you pick is an engineering decision and the requirement is the same, that the words exist in the HTML before any script runs.

Third, make your status codes honest. A missing page should return 404 or 410, and a moved page should return a redirect to its replacement rather than to the homepage. That single change restores the meaning of every monitoring signal you have.

Fourth, put the three checks above on a schedule. Access faults are introduced by deployments, firewall rule changes and vendor defaults, which means a site that passes today can fail next month with nothing in your content calendar to show it.

Where you want the access checks and the content assessment done together, with per-engine results rather than a single blended number, a free visibility assessment runs your buying questions across ChatGPT, Perplexity, Google AI Overviews and Claude and reports what each engine can actually retrieve from your site.

What else do people ask about AI crawlers and empty pages?

Teams ask these five questions most often after discovering that their pages return 200 to everything and get cited by nothing.