Free tool

Can AI assistants read your website?

Check your robots.txt against 22 AI crawlers, from ChatGPT and Gemini to Claude and Perplexity. Free, instant, no signup.

We only read your public robots.txt file.

What this AI crawler checker tests

An AI crawler checker reads a site's robots.txt and reports which AI bots are allowed to fetch it. This one tests 22 crawlers tied to ChatGPT, Claude, Gemini, Copilot, Google AI Mode, Perplexity, Meta AI, Apple, Mistral and DuckDuckGo, plus CCBot and Bytespider.

How the check works

  1. We fetch /robots.txt on the domain you enter and follow its redirects. Each subdomain has its own file, so we read the one you typed.
  2. For each bot, we use the group of rules that names it, or the * group if none does.
  3. We apply the rules of RFC 9309: the longest matching path wins, and Allow wins a tie.
  4. We report whether each bot may fetch your home page. Rules on deeper paths, such as /blog/, are not part of the result.

The 22 AI crawlers we check

Search crawlers decide whether AI engines can cite you, so keep them open. On-demand fetchers open a page when someone asks, and some can ignore robots.txt. Training crawlers collect content for models, and blocking them is your call.

Robots.txt tokenUsed byWhat it does
Search · Required
OAI-SearchBotChatGPTLets ChatGPT show your site in its search answers.
BingbotCopilotBuilds the Bing index that Copilot answers from.
Google-ExtendedGeminiLets Gemini use your pages in its answers. Also controls Gemini training.
GooglebotGoogle AI ModeBuilds the Google index behind AI Mode and AI Overviews.
Claude-SearchBotClaudeIndexes your pages for Claude search results.
PerplexityBotPerplexityShows and links your site in Perplexity results.
ApplebotApplePowers search in Siri, Spotlight and Safari.
meta-webindexerMeta AIHelps Meta AI cite and link to your content.
MistralAI-IndexMistralIndexes your pages for Mistral search.
DuckAssistBotDuckDuckGoReads your pages for DuckDuckGo AI-assisted answers.
On-demand · Optional
ChatGPT-UserChatGPTOpens a page when someone asks ChatGPT. May ignore robots.txt.
Claude-UserClaudeOpens a page when someone asks Claude.
Perplexity-UserPerplexityOpens a page when someone asks Perplexity. Usually ignores robots.txt.
meta-externalfetcherMeta AIOpens a link at a user request. May ignore robots.txt.
MistralAI-UserMistralOpens a page when someone asks Mistral.
Training · Optional
GPTBotChatGPTCollects content to train OpenAI models.
ClaudeBotClaudeCollects content to train Anthropic models.
Applebot-ExtendedAppleControls whether Apple can train its models on your content.
meta-externalagentMeta AITrains Meta AI models and improves Meta products.
MistralAI-TrainingMistralCollects content to train Mistral models.
CCBotCommon CrawlBuilds the Common Crawl public web archive, often used to train AI.
BytespiderByteDanceByteDance crawler, reported to collect AI training data.

Robots.txt rules for AI crawlers

Stay in AI search, opt out of training

OpenAI treats its bots separately, so you can allow OAI-SearchBot and refuse GPTBot.

# Stay visible in AI search
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

Block one AI crawler

User-agent: GPTBot
Disallow: /

A crawler named in a group ignores your * group, so repeat any rule it should still follow. If nothing blocks a bot, it is already allowed and you have nothing to add.

The Google-Extended trap

Blocking Google-Extended is often meant to refuse Gemini training. It also controls whether Gemini Apps can ground their answers in your pages. It does not change your inclusion or ranking in Google Search.

Questions

Last updated November 10, 2026. Sources: OpenAI crawlers · Anthropic crawlers · Perplexity crawlers · Google crawlers · RFC 9309