When you open your website, your browser downloads the HTML, runs the JavaScript, and draws the page. Google’s crawler does much of the same. Most AI crawlers do not: they read the HTML your server sends and stop there.
For many sites that is fine, because the HTML already contains the text. For sites whose content is assembled by JavaScript after the page loads, the AI crawler receives a shell: a title, maybe a menu, and an empty container where your content should be.
Which sites are affected
- Single-page apps built with React, Vue or Angular without server-side rendering
- Some page builders and website templates that load sections with scripts
- Content pulled in after load: reviews widgets, product details, prices, FAQs
Sites built with WordPress, Shopify, Squarespace and most traditional builders usually send real HTML, though widgets and embeds can still hide parts of a page.
How to tell
- Open your page, right-click and choose “View page source” (not “Inspect”). Search for a sentence from your main content. If it is not in the source, AI crawlers probably cannot see it.
- Or run the free scan: its crawler’s eye view shows the exact text an AI crawler receives from your homepage, with a word count.
What to do
- If you use a framework (Next.js, Nuxt, SvelteKit, Astro, Remix), turn on server-side rendering or static generation for public pages. All of them support it.
- If you use a builder, check for a setting that pre-renders pages, and avoid loading key text through embeds.
- At minimum, make sure your homepage HTML contains your business name, what you do, where, for whom, and real links to your main pages.
Why links matter too
Crawlers find your other pages by following links in the HTML. A menu that only exists after JavaScript runs means the crawler never finds your other pages, even if each of them would render fine on its own.