AI crawlers
Definition
AI crawlers are automated programs run by AI companies that fetch web pages, either to gather material for training models or to find content for an answer. Examples include OpenAI's GPTBot, Anthropic's ClaudeBot and Perplexity's PerplexityBot. Google-Extended is a robots.txt control rather than a separate crawler.
Access is managed in robots.txt, with one group of rules per crawler name. Operators publish their crawler names and say they follow those rules, but the file is a request and not a lock. Operators also tend to run more than one crawler, such as one that collects training material and another that fetches a page when a user asks a question, so blocking one name doesn't necessarily block the others. Read each company's own documentation, because names and purposes change.
Whether to block is a business decision, not a default. A blanket rule against 'AI bots', or a block left over from a staging site, can keep pages away from tools that might quote you, while a publisher with paid or proprietary content may reasonably block. Google-Extended is a token you can name in robots.txt to limit how Google uses crawled content for its Gemini models, and Google says it doesn't affect a site's inclusion in Google Search.
A crawler name in a server log proves little, because anyone can claim any name. Where an operator publishes the address ranges its crawlers use, check the log against them before concluding who is visiting.
Definitions are free.So is the cost review.
A short call, then a written review inside 72 hours: what your marketing costs, what it produces, and what is worth fixing first.