Crawlability

Crawlability is a website's ability to be fetched and read by crawlers, from Googlebot and Bingbot to AI crawlers such as OAI-SearchBot and PerplexityBot. A page is crawlable when robots.txt, the CDN or firewall, the server and the HTML all let a bot reach its content. AI engines cannot quote what their crawlers cannot read.

Why crawlability matters for AI search

Crawlability is the first filter of AI visibility: before any ranking or citation, a crawler has to get the page. AI crawlers waste more of their visits than Googlebot does. In Vercel's December 2024 data, 34.82% of ChatGPT's crawler fetches and 34.16% of Claude's hit 404 pages, against 8.22% for Googlebot, and ChatGPT spent another 14.36% following redirects.

Access can also change without anyone editing the site. Cloudflare announced that from September 15, 2026, newly onboarded domains would block Training and Agent bots by default on pages that display ads, and that customers who block training would also block crawlers that mix search and training, such as Googlebot, Applebot and Bingbot.

What makes a page uncrawlable for AI crawlers

A page can fail at five layers, and passing one says nothing about the next:

robots.txt. A Disallow rule for the bot, or for * when the bot has no group of its own.
CDN, firewall or bot protection. A challenge page or a block for unverified bots, even when robots.txt allows them. Google's checklist for its AI features asks that crawling be allowed "in robots.txt, and by any CDN or hosting infrastructure."
Server responses. Errors, timeouts, login walls, broken URLs and redirect chains.
Rendering. Text that only appears after JavaScript runs, which most AI crawlers never do.
Text itself. Google asks that important content be "available in textual form", not only in images or video.

For Google's AI Overviews and AI Mode, one more condition applies: the page must be indexed and eligible to be shown with a snippet.

Example: a page Copilot never cites

Picture a payroll vendor whose comparison page ranks well in Google, yet never shows up when a buyer asks Microsoft Copilot "Which payroll tools suit a UK company with 20 employees?" Copilot answers from Bing's index, so the team checks Bing's crawler. robots.txt allows Bingbot, and the page source contains the full text. The server logs tell the rest: the CDN's bot protection serves Bingbot a challenge page instead of the content. Allowing verified search crawlers in the CDN settings fixes access without touching the site.

How to improve crawlability for AI crawlers

Read robots.txt as each bot that matters to you, and fix accidental blocks.
Review CDN and firewall bot settings after every change of provider or plan.
Check the raw HTML of key pages for the text you want quoted.
Clean up 404s and redirect chains, and keep sitemaps current, as Vercel recommends.
Watch your logs for AI user agents and the status codes they receive.

Hawkscan's action plan covers the first step for you: it flags an AI crawler your robots.txt blocks. For the rendering layer, see JavaScript rendering, and for the rules themselves, robots.txt for AI crawlers.