When we ask GPT-6 Sol which website changes help AI answer engines understand and recommend a product, it goes looking for the crawler rules: on at least three of the seven mornings from October 19 to 25, 2026, its search for that question went to OpenAI's OAI-SearchBot official documentation and the robots.txt rule. So the first change is a robots.txt that lets the search crawlers through, OAI-SearchBot for ChatGPT search and PerplexityBot for Perplexity. Make it, and expect it to get your pages found. On our own site, getting them recommended has been a separate job that the rule did nothing for.
What OpenAI's official OAI-SearchBot documentation says about robots.txt
On October 25 we ran the query GPT-6 Sol issues for this question, "OpenAI OAI-SearchBot official documentation robots.txt", through the search tool GPT-6 Sol uses. It returned 30 results. The first three were OpenAI's own: two help center articles and the crawler overview on its developer site. The publisher FAQ in the help center puts it plainly in the snippet the search returned: "you may need to update your robots.txt file, to ensure OAI-SearchBot has access." OpenAI's comments to South Africa's Competition Commission, also in the list, give the reason it is a rule of its own: the search crawler "operates independently from its crawler for training."
That second line is the one people get wrong. GPTBot collects training data; OAI-SearchBot decides whether your pages can appear in ChatGPT search answers. Blocking GPTBot to stay out of training leaves OAI-SearchBot alone, and the other way round. A Reddit thread in the same 30 results has the mistake as its title: "GPTBot and OAI-SearchBot do different things, and a lot of robots.txt files block the wrong one."
| Crawler | Whose | What it is for | In our robots.txt |
|---|---|---|---|
| OAI-SearchBot | OpenAI | the index behind ChatGPT search | named, allowed |
| ChatGPT-User | OpenAI | fetches a page when a user's chat asks for it | named, allowed |
| GPTBot | OpenAI | training | named, allowed on purpose |
| OAI-AdsBot | OpenAI | ads readiness, per OpenAI's advertiser guidance (the first result on October 25) | not named, so it reads the default group |
| PerplexityBot | Perplexity | Perplexity's own index | named, allowed |
| Claude-SearchBot | Anthropic | the index behind Claude's search | named since September 22, 2026 |
| ClaudeBot, Claude-User, Google-Extended | Anthropic, Google | training, and fetches a user asks for | named, allowed |
Where GPT-6 Sol looks for the PerplexityBot rule
The Perplexity side looks different. On four of the same seven mornings, answering whether AEO carries across ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews, GPT-6 Sol opened Perplexity's crawler guide by its address. Its search does not return that guide. For "PerplexityBot official crawler user agent robots.txt perplexity docs" it gave 32 results on October 25, and Perplexity's own guide was not among them; a post on Perplexity's forum about its crawlers and AWS WAF was second. Ten of the first twelve were reference pages on small sites, one page per crawler, each titled with the crawler's name and its robots.txt rule.
Those reference pages hold the OpenAI query too. Six of its first twelve results were the same kind of page, and three sites (citefiles.com, aiseoexpert.co and trygeo.fast) are in both lists. Two of the six print, in the snippet the search showed, the date they last checked OpenAI's page: October 13 and September 30, 2026. The page that ranks for these queries is narrow and dated: one crawler, its exact token, and when someone last read the source. Two copies of OpenAI's own page ranked as well, a GitHub archive fourth and a mirror domain sixth, so a page that only repeats the source can sit right beside it.
The robots.txt rule that is easy to get wrong
A crawler reads one group of a robots.txt: the group that names it if there is one, and the default User-agent: * group only if there is not. Our file has a default group that keeps crawlers out of the sign-in and admin pages, and a second group that names eight AI crawlers. Once a crawler is named in that second group it stops reading the first, which is why the three Disallow lines are written again inside it. Leave them out and the crawlers you named are the only ones allowed into the pages you meant to close.
Two more details come from the same file. A page you block cannot be read, so a noindex tag on it is never seen and the URL can linger in an index; we leave our customer-only pages open to crawlers and mark them noindex instead. And robots.txt only states permission. A thread in the October 25 results asks why ChatGPT still cannot read a site whose robots.txt allows every AI bot, and suspects the Cloudflare layer in front of it. If a CDN or a bot protection service sits in front of your site, check it on its own, because the file cannot overrule it.
What letting the crawlers in did for us
Our robots.txt lets every crawler in the table through, and our pages are served as plain HTML (the reasoning is on our page about AI crawlers and JavaScript). The October 25 round reads:
| Reading | Result |
|---|---|
| Our tracked pages that GPT-6 Sol's search can find, October 25 | 19 of 23 |
| Results for a search for our own name that are our pages, October 25 | 8 of 15, the first at position 1 |
| The 19 category queries GPT-6 Sol issues for buyer questions that list any page of ours, October 25 | 0 of 19 |
| GPT-6 Sol answers from October 19 to 24 whose own searches returned a page of ours | 0 of 120 |
The last row is 20 buyer questions a morning over six mornings. Being readable got our pages found and our name found. It has not put one page of ours into a category result list, and one of the 4 tracked pages GPT-6 Sol's search cannot find is that page about crawlers and JavaScript.
So, for a buyer asking what to change first: the crawler rule, because it is a few lines in one file and every later change depends on it. After that comes the work the rule cannot do, the words on your pages and the third-party pages that hold your category. The list of what we changed on our site and what each change moved is in website changes that get a product recommended by AI engines.
How we checked
The result lists come from our GPT Visibility round of October 25, 2026: each query run once through the Codex CLI's web search, the search tool GPT-6 Sol uses, in a session of its own. The quotes from OpenAI are the snippets in those results, not pages we fetched separately. What GPT-6 Sol searched comes from our daily AI readouts of October 19 to 25, the same 20 buyer questions once each every morning. The crawler notes are the comments in our own robots.txt as served on October 25. Perplexity's crawler guide is not quoted, because GPT-6 Sol's search did not return it.
Disclosure
ClawWorld is an AEO (Answer Engine Optimization) service for B2B startups based in New York. The measurements are our own, and the zeros are left in.