HawkscanGPTBot
GPTBot is OpenAI's web crawler that collects pages which may be used to train its generative AI foundation models. It is separate from OAI-SearchBot, which makes sites eligible for ChatGPT search, and from ChatGPT-User, which fetches pages when a user asks. Disallowing GPTBot in robots.txt is an opt-out from training only.
Why GPTBot matters
GPTBot is the AI crawler that sites block most often. In Cloudflare's June 2025 sample of the top 10,000 domains, 312 of the 3,816 sites with a robots.txt file disallowed GPTBot, fully or partially, more than any other AI bot. It is also one of the busiest: GPTBot made up 28.1% of AI-only crawler traffic on Cloudflare in July 2025.
Blocking it is a legitimate choice about training data. The mistake is to think it also decides whether ChatGPT can cite you. OpenAI runs three bots for three jobs, and says each setting "is independent of the others."
OAI-SearchBot
OAI-SearchBot is the crawler that decides whether your site can appear in ChatGPT search. According to OpenAI, sites that opt out of it "will not be shown in ChatGPT search answers, though can still appear as navigational links." OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges. After a robots.txt change, OpenAI says it can take about 24 hours for its search systems to adjust. If both GPTBot and OAI-SearchBot are allowed, OpenAI may use one crawl for both purposes.
ChatGPT-User
ChatGPT-User is not a crawler in the usual sense. It visits a page when someone asks ChatGPT or a custom GPT a question that needs it. OpenAI says it "is not used for crawling the web in an automatic fashion" and, because a user starts the request, "robots.txt rules may not apply." It also plays no part in deciding what appears in ChatGPT search: OpenAI points site owners to OAI-SearchBot for that.
Example: opt out of training, stay in ChatGPT search
A software company wants its pages kept out of model training but cited when a buyer asks ChatGPT "Which CRM integrates with Slack for under $20 per user?" Its robots.txt can say so in three groups:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /admin/
Allow: /
User-agent: *
Disallow: /admin/
One detail is easy to miss. Under RFC 9309, a crawler that finds a group with its own name ignores the User-agent: * group. That is why the OAI-SearchBot group repeats the /admin/ rule of the general group instead of relying on it.
How to check your GPTBot rules
Open your site's /robots.txt and look for a group naming GPTBot, OAI-SearchBot or ChatGPT-User, then for a User-agent: * group that would apply to the others. A blanket Disallow: / under * covers every OpenAI bot that has no group of its own. Confirm in your logs that requests claiming to be OpenAI bots come from the IP ranges OpenAI publishes. Hawkscan's action plan flags OAI-SearchBot or ChatGPT-User when your robots.txt blocks them, and leaves GPTBot out of that check: opting out of training is a choice, not an error. See AI crawlers for the equivalent bots at other companies, such as Anthropic's ClaudeBot.
