AI crawlers
Web crawlers used by AI companies to gather training data from websites.
What is AI crawlers?
AI crawlers are automated bots used by AI companies to access and collect web content for training large language models. Examples include GPTBot (OpenAI), ClaudeBot (Anthropic), and Google-Extended (Google AI).
Website owners can choose to block AI crawlers via robots.txt while still allowing search engine crawlers. This is a separate decision from whether to block traditional search crawlers.
Why AI crawlers matters
Blocking AI crawlers prevents your content from being used for AI training, but may reduce your visibility in AI-powered tools. Allowing AI crawlers may help your brand appear in AI responses. This is a strategic decision with tradeoffs.
How to implement
Identify AI crawler user agents
Decide your policy on AI training data
Update robots.txt to allow or block specific crawlers
Monitor for new AI crawler user agents
Consider separate policies for different AI companies
Review your decision as AI search evolves
Code example
# Block all AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Allow search engines
User-agent: Googlebot
Allow: /Common mistakes to avoid
- Blocking all AI crawlers without understanding impact
- Not updating robots.txt as new crawlers emerge
- Assuming blocking crawlers blocks all AI access
- Confusing AI crawlers with search engine crawlers
Examples
GPTBot - OpenAI's crawler for training data
ClaudeBot - Anthropic's crawler
Google-Extended - Google's AI training crawler (separate from Googlebot)
Frequently asked questions
It depends on your goals. Blocking prevents training use but may reduce AI visibility. Many publishers block training crawlers while hoping content in AI responses came from pre-block training.
No, blocking prevents future training data collection. AI models trained before you blocked may still reference your content. Blocking doesn't affect retrieval-augmented systems like Perplexity.