Technical GEO

Technical GEO is the part of generative engine optimization that makes a website reachable and readable for AI systems. It covers crawler access for bots such as OAI-SearchBot and GPTBot, server-side rendering, robots.txt, speed, and indexing in Google and Bing, whose indexes feed Google AI Mode and Microsoft Copilot.

Why technical GEO matters

Technical GEO matters because most AI crawlers read pages differently from Googlebot. In a December 2024 analysis of its network, Vercel found that none of the major AI crawlers it measured rendered JavaScript, including OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot and PerplexityBot. The ChatGPT and Claude crawlers did download JavaScript files (11.50% and 23.84% of their requests) but never executed them. Content that only appears once JavaScript runs is invisible to them.

The gap is common. In April 2026, Adobe reported that product pages on U.S. retail sites scored 66% on average on its AI Content Visibility Checker, meaning about a third of their content could not be read by language models.

What technical GEO covers

Crawler access. OpenAI separates its bots: OAI-SearchBot decides whether a site can appear in ChatGPT search, GPTBot collects training data, and ChatGPT-User fetches pages on a user's request, where "robots.txt rules may not apply". Blocking GPTBot does not hide you from ChatGPT search; blocking OAI-SearchBot does. On Google's side, the Google-Extended token governs Gemini training and grounding, and "does not impact a site's inclusion in Google Search", so blocking it does not remove pages from AI Overviews or AI Mode.
Rendering. Serve main content, prices and navigation in the initial HTML, through server-side rendering or static pages. A price loaded by a client-side script reaches Googlebot, but not GPTBot or ClaudeBot.
Indexing. Google's AI features only link to pages that are indexed and eligible for a snippet, and Copilot answers draw on Bing's index.
Speed and clean URLs. Google's guide recommends a good page experience, including "reducing latency". Vercel saw ChatGPT's crawler spend 34.82% of its fetches on 404 pages, so redirects and sitemaps deserve care.

What technical GEO does not require

Some popular fixes have no proven effect on AI visibility:

llms.txt. Google says Google Search doesn't use it. SE Ranking found no relationship between having the file and AI citations across about 300,000 domains, and in a 90-day test by Otterly, only 84 of more than 62,100 AI bot visits requested it.
Schema markup. Ahrefs tracked 1,885 pages, already cited by AI, that added JSON-LD and found "no major uplift in citations on any platform" across AI Overviews, AI Mode and ChatGPT. Structured data still helps with rich results.
Chunking. Google says there is "no requirement to break your content into tiny pieces" for AI.

How to check your technical GEO

Three checks catch most problems. First, compare your robots.txt and your CDN's bot rules with the user agents each AI company publishes. Second, view your key pages with JavaScript disabled: the text left on screen is close to what most AI crawlers receive. Third, confirm those pages are indexed in Google and Bing. Your server logs then show which AI bots actually visit and which URLs fail.

Hawkscan's action plan flags technical blockers, for example a robots.txt rule that shuts out an AI crawler, and ranks them by impact alongside content actions. For the wider review, see what a GEO audit covers, and how technical work fits into generative engine optimization.