AI Crawler

Why do I need to know this?

AI crawlers can be roughly divided into two groups: training crawlers collect content that may later flow into the training of language models – such as GPTBot or CCBot (Common Crawl). Retrieval crawlers, on the other hand, fetch content live when a user asks an AI assistant a question, for example OAI-SearchBot or PerplexityBot. This distinction matters because the consequences differ: blocking retrieval crawlers removes you from current AI answers; blocking training crawlers mainly limits the use of your content in future models.

Practical relevance for shop and website operators

Shop operators face a trade-off: on the one hand, AI assistants are a growing channel through which customers find products and providers – visibility there requires that the relevant crawlers are allowed to fetch your content. On the other hand, some providers do not want their content to flow into model training, and high-frequency crawlers can create server load. Control is generally exercised via robots.txt, where individual user agents are specifically allowed or excluded. Our GEO optimisation page shows how crawler control fits into a visibility strategy; we address performance questions around crawler load as part of hosting & maintenance.

A common misconception concerns Google: Google-Extended only controls the use of content for Google's AI models (Gemini). Appearing in Google Search and in AI Overviews, however, depends on the regular Googlebot – if you block Google-Extended, you remain visible in search.

Authenticity is a further issue: reputable providers document their crawlers publicly, in some cases including IP ranges, so that access can be verified. Bots that pretend to be well-known crawlers but come from unrelated addresses can be identified this way and blocked at server level – for example via firewall rules or rate limiting.

Typical mistakes

• Blocking all AI crawlers wholesale and thereby unintentionally excluding yourself from AI answers and a growing recommendation channel
• Not distinguishing between training and retrieval crawlers and thus achieving the opposite of what was intended
• Relying blindly on robots.txt – it is a behavioural convention whose compliance is not technically enforced; not every crawler observes it
• Never checking crawler access in the server logs and missing load peaks or unknown bots
• Setting up robots.txt once and never updating it, even though new AI crawlers appear regularly

What to look out for

Make a conscious decision per crawler type and document it in your robots.txt – for example: allow retrieval crawlers in order to be cited in AI answers, and allow or exclude training crawlers depending on your content strategy. Check your server logs regularly for new user agents, and complement crawler control with content measures such as structured data and an llms.txt so that permitted AI systems capture your content correctly.