`. The browser downloads a JavaScript bundle, executes it, fetches data from APIs, and constructs the page in memory. Content appears only after this process completes, sometimes seconds after the initial response.
A human with a modern browser does not notice this. They see a brief loading state, then the finished page.
A crawler is not a human. A crawler issues an HTTP request, receives the response, and parses what it gets. Whether it then executes the JavaScript bundle to discover what the page actually contains is a choice — one made by the crawler operator, weighed against compute cost, latency, and the number of pages they need to index.
Here is how the major crawlers currently behave with respect to JavaScript execution. Treat this as a directional snapshot rather than a contract — these systems change quietly and frequently:
| Crawler | Operator | JavaScript execution | Practical implication for SPAs |
|---|---|---|---|
| Googlebot | Google Search | Yes (deferred, second pass) | Eventually sees content, often with delay |
| Bingbot | Microsoft / Bing | Yes (limited) | Mostly sees content |
| GPTBot | OpenAI / ChatGPT | No reliable execution | Sees the blank shell |
| ClaudeBot | Anthropic | No reliable execution | Sees the blank shell |
| PerplexityBot | Perplexity | Limited | Often sees the blank shell |
| Google-Extended | Google AI / Gemini training | Inherits Googlebot behavior, opt-in | Eventually sees content |
| LinkedInBot | LinkedIn previews | No | Blank shell — broken social previews |
| Slackbot, Twitterbot, FacebookExternalHit | Social previews | No | Blank shell — broken social previews |
The pattern is clear. The traditional search crawlers have invested heavily in JavaScript execution because their commercial model demands it. The newer AI crawlers and the social preview bots have not. They prioritize breadth and freshness over completeness on individual pages, and they assume that any site that genuinely wants to be indexed will return meaningful HTML on the first request.
This is not an oversight. It is a design choice. And it is unlikely to change soon, because executing JavaScript at the scale these crawlers operate at would multiply their infrastructure costs by an order of magnitude.
The takeaway is uncomfortable but precise: if your content is not in the HTML that comes back from the first HTTP request, you should assume AI crawlers cannot see it.
### How to diagnose your own site
Before deciding on a fix, confirm you actually have the problem. Three tests, five minutes total, no tools required beyond a browser and a terminal.
#### Test 1: The curl test
This is the definitive test. It shows you exactly what a crawler sees, with no JavaScript involved.
```bash
curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.0)" \
https://yoursite.com/your-most-important-page \
| head -100
```
A healthy result contains your headline, body copy, structured data, and meta tags in readable HTML. A broken result contains a `
` (often generic), a few meta tags, an empty `` or similar, and one or more `