Hawkscan

Training data

Training data is the body of text, code and other content that a large language model, such as the models behind ChatGPT and Gemini, learns from before it is released. It shapes what an AI assistant can say about a brand from memory, without searching the web, up to the knowledge cutoff of each model.

Why training data matters for your brand

Training data is where a large language model gets its default view of your brand. When ChatGPT or Gemini answers without searching the web, everything it says about you was learned from that data.

Most of it comes from the public web. In the GPT-3 paper, OpenAI reported that 60% of the training mix came from a filtered version of Common Crawl, an open archive of web pages: 410 billion tokens. Quality was weighted too. Wikipedia supplied only 3 billion tokens, but the model saw it 3.4 times during training, against 0.44 times for Common Crawl, because sources judged higher quality were sampled more often.

How training data is collected

AI companies gather training data with web crawlers, alongside curated sets such as books and Wikipedia. Each crawler can be managed in robots.txt, but blocking a training crawler does not always remove a site from AI answers:

  • OpenAI. GPTBot crawls content "that may be used in training" its models. Blocking it does not remove a site from ChatGPT search, which relies on a separate crawler, OAI-SearchBot.
  • Google. The Google-Extended token manages whether content may be used to train future Gemini models and to ground answers in Gemini Apps. Google states that it "does not impact a site's inclusion in Google Search", so AI Overviews and AI Mode are not affected.

Repetition matters. Researchers at OpenAI and Georgia Tech show that facts appearing only once in the training data set a floor on how often a model gets such facts wrong, a common source of AI hallucination.

Knowledge cutoff: where a model's memory stops

The knowledge cutoff is the date after which a model's training data ends. OpenAI lists one for each model: September 30, 2024 for GPT-5 and April 30, 2026 for GPT-6 Astra. A rebrand, a product launch or a price change after that date is unknown to the model.

Example (illustrative): ask an assistant with web search turned off "What is the latest version of Acme Analytics?" It names the last version it saw before its cutoff and presents it as current. Grounding gets around this: Google describes search grounding as a way for Gemini to cite sources "beyond its knowledge cutoff."

How to shape what models learn about you

  • Publish key facts as plain text on crawlable pages, stated the same way everywhere.
  • Earn mentions on independent sites, such as reference sites, reviews and industry media. The more pages state a fact, the more often a model sees it during training.
  • Choose your crawler policy on purpose. Blocking GPTBot tells OpenAI not to use your pages for training; it does not remove them from ChatGPT search.
  • Follow the effect over time. Hawkscan tracks how often ChatGPT, Gemini, Microsoft Copilot and Google AI Mode mention your brand, every day, with your mention rate and visibility score, so a shift after a model update shows up in the trend.

Sources

  1. arXiv (Brown et al., OpenAI), Language Models are Few-Shot Learners ()
  2. OpenAI, Overview of OpenAI Crawlers
  3. Google Search Central, Google's common crawlers ()
  4. OpenAI, GPT-6 Astra model

Your AI visibility starts here.

Your customers are already turning to AI for recommendations. Hawkscan gives you the full picture, so you always know where you stand and where to grow.