Learn · AEO mechanics
AI crawlers don’t run JavaScript — is your site invisible to them?
Most AI crawlers fetch your HTML and leave — they do not execute JavaScript. If your pages render their content client-side, an AI crawler sees a title, an empty shell, and nothing else, no matter how good the rendered page looks in a browser. This single fact explains a large share of “why does no AI engine ever cite us.”
The thirty-second test
Fetch your page the way a crawler does and read what comes back: curl -s https://yoursite.com | grep -i "your product name". If your value proposition, pricing, and FAQ text are not in that raw HTML, they do not exist for AI crawlers. Test your most important pages, not just the homepage — the page you want cited for a buyer question is the page that must pass.
Server-render the words that matter
The fix is to put real content into the HTML at build or serve time: static generation or prerendering for marketing pages, server-side rendering where pages are dynamic. We speak from experience — ClawWorld’s own earlier site was a client-rendered app whose pages were empty to every AI crawler, and rebuilding it as prerendered static HTML was the first item of our own AEO work. The same applies to structured data: JSON-LD injected by a script never reaches a crawler that doesn’t run scripts, so schema must be in the static markup too.
Let the right bots in
Robots rules and bot protection quietly decide your AI visibility. Check three layers: robots.txt actually allows the AI crawlers you want (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended); your CDN or firewall isn’t challenge-blocking them; and every page returns correct status codes — a site whose nonexistent URLs return 200 with an indexable empty page is teaching engines to cache blanks. Verify with a crawler user-agent, for example curl -A "GPTBot" -I https://yoursite.com.
What about llms.txt?
Treat llms.txt as cheap hygiene, not a lever: there is no solid public evidence that engines consult it today, but it costs minutes, is harmless, and states your identity in one crawlable place. Ship it, then spend your real effort on the things engines demonstrably read — static HTML, structured data, and the third-party sources they cite.
Technical floor, not technical ceiling
Crawlability is a pass/fail floor: passing it earns you nothing by itself, but failing it makes every other AEO effort worthless. Once your words are reachable, the game moves to content mechanics — answer units, entity clarity, and presence in the sources engines actually cite.