LanguageAI crawler checker
See which AI crawlers can read your site, from GPTBot and ClaudeBot to PerplexityBot, and the robots.txt line that decides each one.
Which AI crawlers can read your site
Type a domain and we read its public robots.txt, or paste the file. The rules are checked in your browser. No sign-up.
Want to know why AI answers skip your site? Book a free 15-minute call
What to change
Add these lines to robots.txt. A crawler with its own group ignores the * group, so copy any Disallow lines you still want into it.
Sitemap lines
Your crawler check shows here
Type a domain or paste a robots.txt to see which of 12 AI agents can read your homepage, and why.
Your robots.txt decides which AI crawlers may read your site. Most AI companies run separate agents for training, for their search answers and for pages a person asks them to open, and you can allow or block each one.
Blocking a training crawler like GPTBot or ClaudeBot keeps your pages out of future training data. Blocking a search crawler like OAI-SearchBot or PerplexityBot can keep your pages out of the answers those tools give. Blocking Google-Extended doesn’t remove you from Google Search or AI Overviews.
robots.txt is a request, not a lock. The vendors linked below say their crawlers follow it, with a few noted exceptions for user requests.
GPTBot vs OAI-SearchBot
Both belong to OpenAI, and OpenAI’s crawler docs treat them separately. GPTBot collects pages that may be used to train OpenAI’s models. OAI-SearchBot finds pages to show in ChatGPT search. OpenAI says each setting is independent, so you can block one and allow the other.
If you want to stay out of training but still show up in ChatGPT search, block GPTBot and allow OAI-SearchBot. ChatGPT-User is a third agent, used when a person asks ChatGPT to open a page. OpenAI says robots.txt rules may not apply to it, because a person asked for the visit.
Anthropic and Perplexity split their agents the same way: ClaudeBot, Claude-SearchBot and Claude-User, and PerplexityBot and Perplexity-User.
How to allow AI search crawlers in robots.txt
Give the search crawlers their own group with Allow: /. A crawler obeys only the most specific group that names it, so once it has its own group it ignores the * group. Copy any Disallow lines you still want, like /admin, into the new group.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /adminThis lets the three AI search crawlers in, keeps GPTBot out of the whole site and keeps every crawler out of /admin. When there’s a tie between an Allow and a Disallow of the same length, Allow wins. After you deploy the change, run the check again. Crawlers re-read robots.txt from time to time, so the change isn’t instant.
A firewall or CDN bot setting can block AI crawlers even when robots.txt allows them. If your pages still don’t show up, check those settings too.
The AI agents we check
| Token | Company | Type | What it’s for |
|---|---|---|---|
| GPTBot | OpenAI | Training | Training data for OpenAI models |
| OAI-SearchBot | OpenAI | AI search | ChatGPT search results |
| ChatGPT-User | OpenAI | User request | Pages a user asks ChatGPT to open |
| ClaudeBot | Anthropic | Training | Training data for Claude |
| Claude-SearchBot | Anthropic | AI search | Claude search results |
| Claude-User | Anthropic | User request | Pages a user asks Claude to open |
| PerplexityBot | Perplexity | AI search | Perplexity search index |
| Perplexity-User | Perplexity | User request | Pages a user asks Perplexity to open |
| Google-Extended | Control token | Gemini training and grounding control | |
| Applebot-Extended | Apple | Control token | Apple AI training control |
| CCBot | Common Crawl | Training | Open web dataset many models train on |
| meta-externalagent | Meta | Training | Training data for Meta AI models |
Google-Extended and Applebot-Extended never visit your site. They are names you use in robots.txt to say whether Google and Apple may use pages their normal crawlers fetched for AI training.
How the check works
For a domain, our server reads https://yourdomain/robots.txt, follows up to three https redirects and stops after five seconds or 500 KB. It only ever reads that file and /llms.txt at the root of a public domain. The rules are then checked in your browser. A pasted file never leaves your browser.
The rules follow RFC 9309, the robots.txt standard. Each agent obeys the group that names it, or the * group if none does. The longest matching path wins, Allow wins a tie, and * and $ work as wildcards. An empty Disallow blocks nothing. Crawl-delay is ignored.
- No robots.txt (a 404): every crawler may read everything.
- A 401 or 403: the site blocks our reader, so paste the file instead.
- A server error or timeout: we couldn’t read it right now, so paste the file instead.
- An HTML page where robots.txt should be: we treat it as no robots.txt and flag it.
- An empty file: every crawler may read everything.
- A file over 500 KB: we check the first 500 KB and warn you.
FAQ
How do I check if GPTBot is blocked?
Type your domain above and look at the GPTBot row. To check by hand, open yourdomain.com/robots.txt. GPTBot is blocked if a User-agent: GPTBot group has Disallow: /, or if there’s no GPTBot group and the User-agent: * group has Disallow: /.
Is my site blocking ChatGPT?
ChatGPT uses three agents: GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for pages a person asks it to open. Check the OAI-SearchBot and ChatGPT-User rows. This tool only reads robots.txt, so a firewall or CDN bot setting can still block them.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google-Extended is a control token, not a crawler. It decides whether Google can use your pages for Gemini training and grounding. Google says it doesn’t affect your inclusion in Google Search, and AI Overviews are part of Search.
Does llms.txt replace robots.txt?
No. llms.txt is a proposed standard. AI companies haven't confirmed that they use it for answers or rankings. Google's AI optimization guide says an llms.txt file "will neither harm nor help your site's visibility or rankings in Google Search". robots.txt is still the file AI crawlers read to decide what they may fetch.
Read next
- Technical SEO: indexing, rendering and AI crawlers
- How to get cited in AI answers
- How we get startups named in ChatGPT and Perplexity
- Where AI answers get their sources
- llms.txt generator and checker, for building or checking an llms.txt
- SERP preview tool, for checking titles and meta descriptions
Which one sounds like yours?
Your homepage doesn’t say what you do. Said in one line. Help with your copy comes with every full site.
You launch soon with no site. Live in about a week. A launch page, designed and built.
Your site looks like a template. Designed from scratch. Nothing off the shelf.
Visitors can’t find the next step. One clear next step on every page. Book, buy or get in touch.
Your site feels slow. Fast on every screen. Built light and tested on phones.
You can’t edit your own site. Yours to edit. A CMS you can run yourself.
Your site and your brand don’t match. One look everywhere. Brand and site designed together.
People land and leave. A first screen worth staying for. Then a clear way in.
Up close, the details look cheap. Every detail considered. Type, spacing and motion.
You’ve outgrown your first site. A site that fits you now, and grows with you.
