What is robots.txt?

Robots.txt is a plain-text file at the root of a website that tells crawlers which URLs they may fetch, using rules grouped by user agent. Standardized as RFC 9309 in 2022, it is how site owners allow or refuse AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. It is a request, not a lock.

Why robots.txt matters for AI visibility

Robots.txt decides which AI engines may read your site, yet few sites write rules for them on purpose. Cloudflare checked the top 10,000 domains in June 2025: of the 3,816 with a robots.txt file, only 546, about 14%, had rules aimed at AI bots, and disallow rules far outnumbered allow rules. A bot the file doesn't cover, by name or through User-agent: *, is allowed by default.

The stakes are concrete. OpenAI says sites that disallow OAI-SearchBot "will not be shown in ChatGPT search answers." And on Google, AI Overviews and AI Mode follow the same Googlebot rules as the rest of Search.

How robots.txt works

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines, then Allow and Disallow rules for URL paths. Three rules from RFC 9309 catch people out:

A crawler obeys only its own group. If a group names the bot, the bot ignores the User-agent: * group, so general rules must be repeated in each named group.
The most specific rule wins. When Allow and Disallow both match a URL, the longer match applies, and Allow wins a tie.
No match means allowed. A URL that no rule covers can be fetched.

Changes are not instant. OpenAI says its systems can take about 24 hours to adjust to a robots.txt update.

Don't confuse it with llms.txt, a proposed file that points AI agents to pages but allows or blocks nothing.

Example: rules per AI crawler

This file opts out of AI training while keeping the site eligible for AI search, with a private area closed to everyone:

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /account/
Allow: /

User-agent: *
Disallow: /account/

When someone asks ChatGPT search "What's the best invoicing tool for freelancers?", the site stays eligible to be shown, while GPTBot is told its content is not for training. The /account/ rule is repeated in the second group because those bots never read the * group.

What robots.txt does not do

It doesn't enforce anything. RFC 9309 says the rules "are not a form of access authorization." Some fetchers triggered by users, such as ChatGPT-User, may not apply them.
It doesn't keep a page out of Google. Google calls robots.txt "not a mechanism for keeping a web page out of Google": a blocked URL can still appear, without a description. Use noindex instead.
It doesn't override your CDN. A firewall or bot protection can block a crawler your file allows.
It doesn't remove past data. A new Disallow governs future crawling, not content a company already collected.

How to check your robots.txt for AI crawlers

Open yoursite.com/robots.txt and read it once for each bot that matters to you: find its group, or the * group if it has none, and check the paths you want cited. Repeat for each subdomain, since every host has its own file. Hawkscan's action plan reads your robots.txt and flags an AI answer crawler it blocks, such as OAI-SearchBot for ChatGPT search or Bingbot for Microsoft Copilot. For a map of who runs which bot, see AI crawlers, and for the OpenAI case in detail, GPTBot.