# AI crawlers in robots.txt: which to allow and which to block

GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended explained: what each does and the robots.txt lines to allow search, not training.

Canonical: https://ignura.com/blog/ai-crawlers-robots-txt
Last updated: 2026-10-29
Author: Jawad Jalal

If you want to show up in AI answers, allow the AI search crawlers (OAI-SearchBot, Claude-SearchBot and PerplexityBot) in robots.txt. If you don't want your pages used for model training, block the training crawlers (GPTBot and ClaudeBot) and the Google-Extended token. The two choices are separate, so you can do both.

Then check that your CDN or firewall isn't blocking the bots that robots.txt allows. The free [AI crawler checker](https://ignura.com/tools/ai-crawler-checker) reads your robots.txt and shows what it says to 12 AI crawlers, one line each.

## The short version

- Most AI companies run separate bots for training, for their search answers and for pages a user asks them to open. You can allow or block each one.
- Blocking a search bot can keep you out of that tool's answers. OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers", though they can still appear as navigational links ([OpenAI](https://developers.openai.com/api/docs/bots)).
- Blocking a training bot tells that company not to use your pages for training. It doesn't remove you from its search answers.
- Google-Extended does not affect Google Search. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" ([Google](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)). AI Overviews and AI Mode use Googlebot.
- robots.txt is a request, not a lock. Bots fetching a page for a user may not follow it, and a firewall can block a bot that robots.txt allows.

## Which AI crawler does what

Each line below was checked against the company's own crawler docs on 28 October 2026.

| User agent | Company | Type | What it's for |
| --- | --- | --- | --- |
| GPTBot | OpenAI | Training | Crawls content that may be used to train OpenAI's models ([docs](https://developers.openai.com/api/docs/bots)) |
| OAI-SearchBot | OpenAI | AI search | Surfaces sites in ChatGPT search ([docs](https://developers.openai.com/api/docs/bots)) |
| ChatGPT-User | OpenAI | User request | Visits a page when a user asks ChatGPT something ([docs](https://developers.openai.com/api/docs/bots)) |
| ClaudeBot | Anthropic | Training | Collects content that could go into training ([docs](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) |
| Claude-SearchBot | Anthropic | AI search | Indexes content to improve Claude's search results ([docs](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) |
| Claude-User | Anthropic | User request | Fetches a page when a user asks Claude a question ([docs](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) |
| PerplexityBot | Perplexity | AI search | Surfaces and links sites in Perplexity search, not used for training ([docs](https://docs.perplexity.ai/guides/bots)) |
| Perplexity-User | Perplexity | User request | Visits a page to answer a user's question ([docs](https://docs.perplexity.ai/guides/bots)) |
| Googlebot | Google | Search | Crawls for Google Search, including AI Overviews and AI Mode ([docs](https://developers.google.com/search/docs/appearance/ai-features)) |
| Google-Extended | Google | Control token | Says whether Google may use your content for Gemini training and grounding ([docs](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)) |

Three things in that table trip people up.

**User-request bots may skip robots.txt.** OpenAI says that because ChatGPT-User visits are started by a user, "robots.txt rules may not apply". Perplexity says Perplexity-User "generally ignores robots.txt rules" for the same reason. Anthropic is the exception: it says disabling Claude-User stops Claude from fetching your pages for user questions.

**Google-Extended never visits your site.** It is a name you use in robots.txt. Google's normal crawlers do the fetching, and the token says whether Google may use what they fetched for Gemini. To limit what Google shows from your pages in Search, including AI features, Google points to nosnippet, data-nosnippet, max-snippet and noindex instead ([Google](https://developers.google.com/search/docs/appearance/ai-features)).

**Changes take time.** OpenAI and Perplexity both say it can take about 24 hours for their systems to pick up a robots.txt change.

The checker also covers Applebot-Extended, CCBot and meta-externalagent. Its page lists every user agent it checks with a link to each company's docs.

## robots.txt to show up in AI answers

This lets every crawler in, AI bots included, and keeps them all out of private paths. If your current file has no AI rules and nothing blocks them upstream, you may already be here.

```
User-agent: *
Disallow: /admin
Disallow: /account
```

If your file blocks bots broadly and you want the AI search crawlers back in, give them their own group:

```
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin

User-agent: *
Disallow: /
```

A crawler follows the group that names it and ignores the `*` group, so copy any Disallow lines you still want into the new group. That rule comes from [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), the robots.txt standard: crawlers obey the group that matches their name, and fall back to `*` only when none does.

## robots.txt to block training only

This keeps your pages out of OpenAI's and Anthropic's training crawls and opts out of Gemini training and grounding, while search and user bots can still read the site:

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /admin
```

OpenAI says each setting is independent: you can block GPTBot and still allow OAI-SearchBot. Perplexity says PerplexityBot is not used to crawl content for AI foundation models, so there is no separate Perplexity training bot to block.

Anthropic asks you to add the rules to robots.txt on every subdomain you want to opt out.

## CDN and firewall blocks that override robots.txt

robots.txt says what a bot may fetch. Your CDN, firewall or hosting can still turn the request away before it reaches the file. When that happens, robots.txt says yes and the bot gets a 403 or a challenge page.

- **Cloudflare.** Since 1 July 2025, Cloudflare blocks AI crawlers by default and asks every new domain at sign-up whether to allow them ([Cloudflare](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/)). Check your AI crawler and bot settings if your site sits behind it.
- **Other firewalls.** Perplexity says you may need to allow its bots explicitly in a web application firewall, and gives steps for Cloudflare and AWS WAF ([Perplexity](https://docs.perplexity.ai/guides/bots)).
- **Match on IP ranges, not just the name.** OpenAI and Perplexity publish IP ranges for each bot. An allow rule on user agent and IP together is harder to fake than one on the user agent alone.
- **Don't use IP blocks as your opt-out.** Anthropic says blocking its IP addresses "may not work correctly or persistently guarantee an opt-out", because it stops Anthropic reading your robots.txt. Use robots.txt rules instead.

Google's own guide makes the wider point: robots.txt rules "cannot enforce crawler behavior", and it is not a way to keep a page out of Google. Use noindex or a password for that ([Google](https://developers.google.com/search/docs/crawling-indexing/robots/intro)).

## Where llms.txt fits

llms.txt doesn't allow or block anything. It is a Markdown map of your site for agents. Google says the file "will neither harm nor help your site's visibility or rankings in Google Search" ([Google](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)). No major AI company has said its crawlers read the one on your site. If GPTBot or PerplexityBot is blocked in robots.txt, an llms.txt won't let it in.

Get robots.txt right first. If you also want an llms.txt, the [llms.txt generator](https://ignura.com/tools/llms-txt-generator) drafts one from your sitemap, and [What is llms.txt](https://ignura.com/blog/what-is-llms-txt) explains what it does and doesn't do.

## How to check what AI crawlers can see

1. **Run the checker.** Put your domain into the [AI crawler checker](https://ignura.com/tools/ai-crawler-checker). It reads your live robots.txt and shows, for each of 12 AI crawlers, whether it may read your site and which line decides it. You can also paste a draft file to test it before you deploy.
2. **Test a real page, not just the homepage.** A rule like `Disallow: /blog` can hide the pages you most want quoted. Check the paths for your docs, blog and comparison pages.
3. **Look for blocks upstream.** If robots.txt allows a bot but its requests fail, check your CDN and firewall logs for 403s and challenges against its user agent.
4. **Read your server logs.** Search for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot. If a search bot that you allow never shows up, something upstream may be stopping it.
5. **Check every subdomain.** robots.txt applies per host. docs.example.com needs its own file.
6. **Recheck after changes.** Give it a day for OpenAI and Perplexity to pick up the new rules.

Getting crawled is the first step, not the last. Once the bots can read your pages, the [guide to getting cited in AI answers](https://ignura.com/blog/how-to-get-cited-in-ai-answers) covers what makes a page worth quoting.

## FAQ

### Should I block GPTBot?

Only if you don't want your content used to train OpenAI's models. Blocking GPTBot doesn't remove you from ChatGPT search. That is OAI-SearchBot, which OpenAI controls separately.

### What is the difference between GPTBot and OAI-SearchBot?

GPTBot crawls content that may be used to train OpenAI's models. OAI-SearchBot finds pages to show in ChatGPT search. OpenAI says the two settings are independent, so you can block one and allow the other.

### Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode use pages Googlebot crawls. To limit what Google shows from a page, use nosnippet, max-snippet or noindex.

### Do AI crawlers follow robots.txt?

The crawlers from OpenAI, Anthropic, Perplexity and Google say they do. Bots that fetch a page because a user asked, such as ChatGPT-User and Perplexity-User, may not. robots.txt is a request, so a firewall is the only way to enforce a block.

### Can llms.txt allow or block AI crawlers?

No. Allowing and blocking happens in robots.txt and at your CDN or firewall. llms.txt is an optional map of your site for agents.

### How long does a robots.txt change take to work?

OpenAI and Perplexity say about 24 hours. Other crawlers re-read the file on their own schedule, so give it a few days before you judge the result.

## Not sure what AI crawlers see on your site?

Crawler access is one part of the [technical layer](https://ignura.com/technical-seo). The bigger question is whether ChatGPT, Perplexity and Google's AI name you when buyers ask about your category. How we price that work is on the [pricing page](https://ignura.com/pricing).

If you want a second opinion on your robots.txt and firewall settings, we'll go through them with you on a free 15-minute call.

[Book your free Growth Check](https://ignura.com/book?utm_source=site&utm_medium=blog&utm_content=ai-crawlers-robots-txt)

Book a 15-minute call about your website: https://ignura.com/book