On this page
Key takeaways
- AI companies now run separate crawlers for training, for search, and for visits a user asks for. Blocking one does not block the others.
- To be found in AI search answers, allow the search crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- Blocking training crawlers such as GPTBot or ClaudeBot is a business choice. It does not remove you from AI search.
- Google-Extended controls use of your content for Gemini training and grounding. It does not affect Google Search.
Why this matters#
Your robots.txt file tells crawlers which parts of your site they may visit. Many sites added blanket blocks for "AI bots" in 2023 and 2024. Since then, the main AI companies have split their crawlers by purpose, and an old block may be stopping AI search tools from reading your pages, which makes it harder for them to name you.
The main AI crawlers#
| Crawler | Company | What it does |
|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages to show in ChatGPT search features |
| GPTBot | OpenAI | Collects content that may be used to train OpenAI's models |
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user's request needs it |
| Claude-SearchBot | Anthropic | Indexes content to improve Claude's search answers |
| ClaudeBot | Anthropic | Collects content that may be used for model training |
| Claude-User | Anthropic | Visits a page when a Claude user's request needs it |
| PerplexityBot | Perplexity | Finds and links pages in Perplexity answers; Perplexity says it is not used for model training |
| Perplexity-User | Perplexity | Visits a page when a user's question needs it |
| Google-Extended | A control (not a separate crawler) for use of your content in Gemini training and grounding |
A few details worth knowing, from each company's own documentation:
- OpenAI says that because ChatGPT-User actions are started by a user, robots.txt rules may not apply to them.
- Perplexity says Perplexity-User generally ignores robots.txt, for the same reason.
- Anthropic says its bots honour robots.txt, and that blocking Claude-User or Claude-SearchBot may reduce your visibility in Claude's search answers.
- Google says Google-Extended does not affect your inclusion or ranking in Google Search.
A sample robots.txt for AI visibility#
This allows AI search crawlers and leaves training to your choice:
# AI search: allow, so AI tools can find and cite your pages
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# AI training: your choice. Change Allow to Disallow to opt out.
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /Keep your normal rules for private areas, such as a login or admin section, for all crawlers.
Should you allow training crawlers?#
There is no single right answer. Allowing them means future models may learn from your public pages, which can help AI describe your company from memory. Blocking them keeps your content out of future training sets. For most B2B companies whose public pages are marketing pages, allowing them is the simpler choice. Companies with paid content or original research may decide differently.
How to check your site#
- Open yourdomain.com/robots.txt in a browser.
- Look for lines like
User-agent: GPTBotfollowed byDisallow: /, or a rule forUser-agent: *that blocks everything. - Check that your firewall or bot protection is not blocking these crawlers too. Some security services block AI crawlers by default, regardless of robots.txt.
Then read our AEO starter checklist for the rest of the technical basics.
Sources#
Questions and answers
Will blocking GPTBot remove my site from ChatGPT?
Not from ChatGPT search. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot. To appear in ChatGPT search answers, allow OAI-SearchBot.
Does blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews in Search are covered by Googlebot, not Google-Extended.
Do AI crawlers obey robots.txt?
The main AI companies say their automatic crawlers follow robots.txt. Visits made because a user asked for a page, such as ChatGPT-User and Perplexity-User, may not.
B2B marketers who build tools to measure how ChatGPT, Gemini, Claude and Perplexity talk about companies. About us