Why many AI crawlers cannot see your site

One HTTP request, read what the server returns, move to the next address. That is all. It does not open a browser, does not wait for scripts to finish, does not click anything. Whatever is not in the server's first response does not exist to it.

That is a substantial difference from Googlebot, which has run a rendering service for years: it fetches the page, queues it, executes it in a headless browser, then indexes the result. With some delay and some limits, but it executes. The habit built on that behaviour is what misleads now: if a page works for Google, people assume it works for everyone.

The Vercel and MERJ analysis across more than five hundred million requests, published in December 2024, is the widest measurement available on this. The result is unambiguous, and should be read with its scope: zero evidence of JavaScript execution in the crawlers of OpenAI, Anthropic, Meta, ByteDance, Perplexity and Common Crawl. They do fetch JavaScript files, in a non-trivial share of requests, but never run them. Googlebot and the systems reusing its infrastructure sat outside that claim, and still do. None of the named vendors has announced otherwise since, but this is a capability that can change without notice: the date matters.

The split does not follow the logic you would expect. It is not about how large the vendor is, it is about which infrastructure they reused. Whoever already ran a search engine with a rendering service is using it; whoever built a crawler from scratch to train models is not.

Not the decorative content. What almost always disappears is the deciding information, because that is the part loaded later, often from an external service, for performance or management reasons.

You need one thing: to look at the page the way something that executes nothing sees it. The most direct way is to fetch the HTML from a terminal and search inside it for the text you care about. If `curl` is unfamiliar, disabling JavaScript for the site in your browser settings and reloading works too, as does View Page Source, which shows the served document rather than the reconstructed one.

Watch for one trap: developer tools show the DOM after scripts have run, so the price is always visible there. That is the wrong view. The right view is the response source, not the inspector.

Because they discover badly. In the same study, more than a third of ChatGPT's and Claude's fetches land on pages that do not exist, against roughly eight per cent for Googlebot. A share also follows redirect chains. These are crawlers that guess addresses more than they derive them.

The practical consequence is counterintuitive and useful: with these systems discovery matters more than optimisation. An up-to-date sitemap, text-based internal links, stable URLs, and direct redirects are worth more than any fine-grained content work, because they decide whether the page is reached at all.

A rule about redirects follows. Every chain is a point where a crawler that follows few of them stops. If you migrated the site and left two or three hops between the old address and the new one, you are paying for it here.

Do not rewrite it. That is the most expensive reaction and almost always unnecessary, because the problem does not affect the whole site equally: it affects the pages that need to be read by an external system. The rest can stay as they are.

The minimum intervention is making content available in the server response for those pages: server-side rendering, static generation, or a prerendering service in front of the site. Which of the three depends on your stack, not on the problem. There is one criterion: after the work, the text that matters must be in the served HTML.

The right order is by commercial value, not technical ease. Product pages and categories first, service pages and source content next, the rest when there is room. Ten pages done properly beats a migration project that never finishes.

In `robots.txt`, and the decision should be made per group rather than in bulk, because AI crawlers do not all do the same thing. Search crawlers serve real-time answers and can cite the source; training crawlers collect material to build models and cite nothing immediately. Two different choices, with two different consequences.

The most expensive confusion involves OpenAI, because it uses two agents with similar names. `OAI-SearchBot` serves search inside ChatGPT; `GPTBot` serves training. Blocking `GPTBot` does not remove you from search inside ChatGPT, and blocking `OAI-SearchBot` does. Anyone copying a generic block found online often gets the exact opposite of what they wanted: they stay in the training data and disappear from answers that carry a citation.

Then there is a category of blocks nobody decided. Many web application firewalls and protection services answer unrecognised user agents with a security challenge, and AI crawlers do not solve challenges. The site is open in `robots.txt` and closed in practice, which is the worst situation because it shows up nowhere. It is worth checking the logs for the declared user agents and verifying which status code they actually receive.

Whatever the choice, write it in the file as a comment, per group. A line saying why that group is allowed or blocked keeps the decision reversible in six months, when conditions have changed and nobody remembers the reasoning.

It does not make you citable. It makes you legible, which is necessary and not sufficient. An engine that can now read your page may still not pick you as a source, because that decision depends on authority and consistency, not on accessibility.

It does not compensate for badly structured content. If the answer exists but is spread across six paragraphs that have to be read in sequence, rendering made it available and not liftable. Those are two different problems with two different pieces of work.

It does not guarantee crawlers will come. You can be perfectly legible and still see few crawler visits, because frequency depends on how known and linked your domain already is. Rendering is the first step, not the staircase.