Can AI assistants read your website?
Check your robots.txt against 22 AI crawlers, from ChatGPT and Gemini to Claude and Perplexity. Free, instant, no signup.
What this AI crawler checker tests
An AI crawler checker reads a site's robots.txt and reports which AI bots are allowed to fetch it. This one tests 22 crawlers tied to ChatGPT, Claude, Gemini, Copilot, Google AI Mode, Perplexity, Meta AI, Apple, Mistral and DuckDuckGo, plus CCBot and Bytespider.
How the check works
- We fetch /robots.txt on the domain you enter and follow its redirects. Each subdomain has its own file, so we read the one you typed.
- For each bot, we use the group of rules that names it, or the * group if none does.
- We apply the rules of RFC 9309: the longest matching path wins, and Allow wins a tie.
- We report whether each bot may fetch your home page. Rules on deeper paths, such as /blog/, are not part of the result.
The 22 AI crawlers we check
Search crawlers decide whether AI engines can cite you, so keep them open. On-demand fetchers open a page when someone asks, and some can ignore robots.txt. Training crawlers collect content for models, and blocking them is your call.
| Robots.txt token | Used by | What it does |
|---|---|---|
| Search · Required | ||
| OAI-SearchBot | ChatGPT | Lets ChatGPT show your site in its search answers. |
| Bingbot | Copilot | Builds the Bing index that Copilot answers from. |
| Google-Extended | Gemini | Lets Gemini use your pages in its answers. Also controls Gemini training. |
| Googlebot | Google AI Mode | Builds the Google index behind AI Mode and AI Overviews. |
| Claude-SearchBot | Claude | Indexes your pages for Claude search results. |
| PerplexityBot | Perplexity | Shows and links your site in Perplexity results. |
| Applebot | Apple | Powers search in Siri, Spotlight and Safari. |
| meta-webindexer | Meta AI | Helps Meta AI cite and link to your content. |
| MistralAI-Index | Mistral | Indexes your pages for Mistral search. |
| DuckAssistBot | DuckDuckGo | Reads your pages for DuckDuckGo AI-assisted answers. |
| On-demand · Optional | ||
| ChatGPT-User | ChatGPT | Opens a page when someone asks ChatGPT. May ignore robots.txt. |
| Claude-User | Claude | Opens a page when someone asks Claude. |
| Perplexity-User | Perplexity | Opens a page when someone asks Perplexity. Usually ignores robots.txt. |
| meta-externalfetcher | Meta AI | Opens a link at a user request. May ignore robots.txt. |
| MistralAI-User | Mistral | Opens a page when someone asks Mistral. |
| Training · Optional | ||
| GPTBot | ChatGPT | Collects content to train OpenAI models. |
| ClaudeBot | Claude | Collects content to train Anthropic models. |
| Applebot-Extended | Apple | Controls whether Apple can train its models on your content. |
| meta-externalagent | Meta AI | Trains Meta AI models and improves Meta products. |
| MistralAI-Training | Mistral | Collects content to train Mistral models. |
| CCBot | Common Crawl | Builds the Common Crawl public web archive, often used to train AI. |
| Bytespider | ByteDance | ByteDance crawler, reported to collect AI training data. |
Robots.txt rules for AI crawlers
Stay in AI search, opt out of training
OpenAI treats its bots separately, so you can allow OAI-SearchBot and refuse GPTBot.
# Stay visible in AI search User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / # Opt out of model training User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot Disallow: /
Block one AI crawler
User-agent: GPTBot Disallow: /
A crawler named in a group ignores your * group, so repeat any rule it should still follow. If nothing blocks a bot, it is already allowed and you have nothing to add.
The Google-Extended trap
Blocking Google-Extended is often meant to refuse Gemini training. It also controls whether Gemini Apps can ground their answers in your pages. It does not change your inclusion or ranking in Google Search.
Questions
Last updated November 10, 2026. Sources: OpenAI crawlers · Anthropic crawlers · Perplexity crawlers · Google crawlers · RFC 9309