AI crawlers

How AI crawlers work

AI crawlers fall into three families, and one company often runs one of each:

• Training crawlers collect pages that may be used to train future models. Disallowing them is an opt-out from training, not from answers.
• Search crawlers build the index an AI engine searches when it answers, the way Googlebot does for Google Search.
• User-triggered fetchers visit a page live because a user's question needs it. They don't crawl on their own, and some don't apply robots.txt.

Most of them announce themselves with a user agent name and follow the robots.txt rules addressed to that name. The main ones, from each company's documentation:

Two limits apply. Rules are a request, not a lock: the robots.txt standard, RFC 9309, says they "are not a form of access authorization." And a CDN or firewall can block a crawler that robots.txt allows.

PerplexityBot

PerplexityBot is Perplexity's search crawler. Perplexity says it is "designed to surface and link websites in search results on Perplexity" and is "not used to crawl content for AI foundation models", and it recommends allowing it for a site to appear in those results. Its companion, Perplexity-User, fetches pages when someone asks Perplexity a question. Because a user requested the fetch, Perplexity says it "generally ignores robots.txt rules." Changes to robots.txt can take up to 24 hours to apply, and sites behind a web application firewall may also need to allow Perplexity's published IP ranges.

Google-Extended

Google-Extended is not a crawler but a robots.txt token. Google crawls with its usual user agents and reads Google-Extended as a usage preference: whether content it crawls may be used to train future Gemini models and to ground answers in Gemini Apps and in Grounding with Google Search on Vertex AI. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

So disallowing Google-Extended does not remove a site from AI Overviews or AI Mode. Those are part of Search, governed by your Googlebot rules and by snippet controls such as nosnippet and noindex. It can, however, stop Gemini Apps from using your pages to ground their answers.

Example: one robots.txt, four engines

Picture a project management SaaS whose robots.txt contains a block copied from a "block AI bots" list:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Disallow: /

A buyer then asks: "What's the best project management tool for a remote team of 15?" Based on each company's documentation, here is what that file does:

• ChatGPT search won't show the site in its answers, because OAI-SearchBot is blocked. The site can still appear as a navigational link.
• Perplexity loses access through its search crawler, the one it recommends allowing to appear in its results.
• Gemini Apps can no longer ground their answers on the site's pages.
• Google AI Mode and AI Overviews are unaffected, because Googlebot is still allowed.

The owner only wanted to opt out of training. Rules for GPTBot, ClaudeBot and Google-Extended would have done that without touching ChatGPT search or Perplexity, and only the Google-Extended line costs anything in answers, in Gemini Apps.

How to audit your AI crawler access

1. Read your robots.txt as each bot. For every crawler in the table, find the group that applies to it. A bot with its own group ignores the User-agent: * group entirely.
2. Separate training from search. Disallow training crawlers if you choose to, and allow the search crawlers of the engines you want to appear in.
3. Check the layer below. Google's checklist for its AI features asks that crawling be allowed "in robots.txt, and by any CDN or hosting infrastructure."
4. Confirm in your logs. Look for each user agent and verify its IP address against the ranges the company publishes.

Hawkscan's action plan flags an AI crawler your site blocks, such as OAI-SearchBot for ChatGPT search or Bingbot for Microsoft Copilot, among the fixes it ranks by impact. For the rules themselves, see robots.txt for AI crawlers.

Common questions

Should I block AI crawlers?

It depends on what you want to stop. Blocking training crawlers such as GPTBot or ClaudeBot keeps your content out of future model training without removing you from those companies' search results. Blocking search crawlers such as OAI-SearchBot or PerplexityBot takes you out of those engines' answers. Decide crawler by crawler, not with one rule for every "AI bot".

Do AI crawlers respect robots.txt?

The major ones say their automatic crawlers do: OpenAI, Anthropic, Perplexity and Google all document robots.txt controls. User-triggered fetchers are the exception. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores" them, because a person asked for the page.

How can I see which AI crawlers visit my site?

Search your server or CDN logs for the user agent names in the table above. User agents are easy to fake, so check the IP address against the lists each company publishes, such as OpenAI's searchbot.json or Perplexity's perplexitybot.json. Bing offers a Verify Bingbot tool for the same purpose.