Is robots.txt Blocking AI on Your Site?

Learn how your robots.txt file may be preventing AI assistants from accessing and citing your content.

Markdown anzeigenChatGPT fragen Claude fragen Perplexity fragen Copilot fragen

If you're seeing this page, our analysis detected that your robots.txt file is blocking crawlers from accessing your site. This may be preventing both our analysis tool and AI assistants like ChatGPT, Claude, and Perplexity from reading your content.

The Problem: Overly Restrictive robots.txt

Your robots.txt file tells crawlers which parts of your site they're allowed to access. However, if configured too restrictively, it can block legitimate AI assistants from accessing your content.

Common blocking patterns we detected:

Wildcard Disallow: Disallow: / under User-agent: *
Specific AI Bot Blocking: Disallow rules for GPTBot, ClaudeBot, etc.
Spyglasses Blocking: Disallow rules specifically for our crawler

Why This Matters

What a block does depends on which kind of AI bot you block. Spyglasses sorts AI bots into three roles:

Role	Examples	What they do	If you block them
AI Visitors	`ChatGPT-User`, `Claude-User`, `Perplexity-User`	Open a page live while an AI assistant writes an answer.	The assistant can't open your page live. It has to work from what it already knows about the page.
AI Search Indexers	`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`	Add pages to an AI assistant's own search index.	Your pages may be left out of that assistant's search results. Assistants also find pages in other ways, so your site may still be cited.
AI Model Trainers	`GPTBot`, `ClaudeBot`, `Google-Extended`, `CCBot`	Collect pages to train future AI models.	Future AI models may learn less from your site. Today's answers aren't affected.

Unlike malicious crawlers, AI crawlers from major companies respect robots.txt, so a rule you write for one of them is followed. That makes it worth checking which roles your rules actually block.

How to Check Your robots.txt

View Your Current File

Visit: https://yoursite.com/robots.txt

Look for problematic patterns:

❌ Problem: Blocking Everything

User-agent: *
Disallow: /

This blocks all crawlers from all pages. Unless you intentionally want to hide your site from all search engines and AI, this is too restrictive.

❌ Problem: Blocking AI Assistants

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

This blocks OpenAI's and Anthropic's training crawlers and Perplexity's search crawler. Check the three roles above to see what each of these blocks does.

❌ Problem: Blocking Analysis Tools

User-agent: Spyglasses
Disallow: /

This prevents us (and similar tools) from analyzing your AI visibility.

How to Fix It

Option 1: Allow All Crawlers (Recommended)

If you don't have sensitive content to protect:

User-agent: *
Allow: /

Or simply have an empty robots.txt or no file at all—this allows everything.

Option 2: Selectively Allow AI Assistants

If you need to block some crawlers but want AI visibility:

# Allow major AI assistants
User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Spyglasses
Allow: /

# Block others
User-agent: *
Disallow: /

Note: Allow directives must come before Disallow for the same user agent.

Option 3: Allow Most, Block Sensitive Areas

If you have specific pages to protect:

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /internal/
Allow: /

This allows access to public content while protecting sensitive areas.

AI Crawler User Agents to Allow

Here are the major AI crawlers you should consider allowing, with the role each one plays:

AI Assistant	User Agent	Role	Company
ChatGPT	`ChatGPT-User`	AI Visitor	OpenAI
ChatGPT	`OAI-SearchBot`	AI Search Indexer	OpenAI
ChatGPT	`GPTBot`	AI Model Trainer	OpenAI
Claude	`Claude-User`	AI Visitor	Anthropic
Claude	`Claude-SearchBot`	AI Search Indexer	Anthropic
Claude	`ClaudeBot`	AI Model Trainer	Anthropic
Claude (older name)	`anthropic-ai`	AI Model Trainer	Anthropic
Perplexity	`Perplexity-User`	AI Visitor	Perplexity
Perplexity	`PerplexityBot`	AI Search Indexer	Perplexity
Google Gemini	`Google-Extended`	AI Model Trainer (robots.txt only)	Google
Bing Copilot	`Bingbot`	Search engine crawler	Microsoft
Spyglasses	`Spyglasses`	Analysis	Spyglasses

Comprehensive AI Allowlist

# OpenAI (ChatGPT)
User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

# Anthropic (Claude)
User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

# Perplexity
User-agent: Perplexity-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Google (Gemini, AI Overviews)
User-agent: Google-Extended
Allow: /

User-agent: Googlebot
Allow: /

# Microsoft (Copilot)
User-agent: Bingbot
Allow: /

# Spyglasses
User-agent: Spyglasses
Allow: /

# Allow general crawlers
User-agent: *
Allow: /

Testing Your robots.txt

1. Validate Syntax

Use Google's robots.txt Tester:

Visit Google Search Console
Go to robots.txt Tester
Enter your robots.txt content
Test specific user agents and URLs

2. Test Specific User Agents

Use our testing tool or manually test:

# Test if GPTBot is allowed
curl -A "GPTBot" https://yoursite.com/robots.txt
 
# Test a specific page
curl -A "GPTBot" https://yoursite.com/about

3. Request a New Analysis

After updating your robots.txt:

Return to Spyglasses AI Visibility Report
Request a new analysis
We'll verify that crawlers can now access your site

Common Scenarios

E-commerce Sites

User-agent: *
# Block cart and checkout (private)
Disallow: /cart/
Disallow: /checkout/
# Block search results (duplicate content)
Disallow: /search
# Allow everything else including product pages
Allow: /

SaaS/Software Companies

User-agent: *
# Block admin and user dashboards
Disallow: /app/
Disallow: /dashboard/
# Allow public marketing pages, docs, blog
Allow: /

Content/Media Sites

User-agent: *
# Allow everything - you want maximum visibility
Allow: /

Local Businesses

User-agent: *
# Allow everything - AI citations drive local discovery
Allow: /

Best Practices

Do's

✅ Do allow AI assistants unless you have a specific reason not to
✅ Do use specific Disallow rules for sensitive areas
✅ Do test your robots.txt after changes
✅ Do monitor crawler behavior in your analytics
✅ Do keep your robots.txt simple and maintainable

Don'ts

❌ Don't use Disallow: / under User-agent: * unless intentional
❌ Don't block AI crawlers without understanding the impact
❌ Don't use robots.txt for access control (use authentication instead)
❌ Don't block content you want to be discovered
❌ Don't copy robots.txt from other sites without understanding it

Understanding Crawler Respect Levels

Not all crawlers respect robots.txt equally:

Crawler Type	Respects robots.txt
Major AI Assistants (GPT, Claude, etc.)	✅ Yes, strictly
Search Engines (Google, Bing)	✅ Yes, strictly
Malicious Bots	❌ Often ignore it
Aggressive Scrapers	❌ Often ignore it

Key Insight: The crawlers you want to access your site (AI assistants, search engines) respect robots.txt. Malicious crawlers often ignore it. Blocking legitimate crawlers only hurts your visibility.

What About AI Training?

Some companies have added robots.txt rules to prevent AI training:

User-agent: GPTBot
Disallow: /

Important distinction: OpenAI uses a different crawler for each role.

GPTBot: collects pages to train future models
OAI-SearchBot: builds ChatGPT's own search index
ChatGPT-User: opens pages live while ChatGPT answers (not training)

Anthropic follows the same pattern with ClaudeBot, Claude-SearchBot and Claude-User.

If you want to block training but stay available to ChatGPT:

# Block training
User-agent: GPTBot
Disallow: /

# Allow ChatGPT search and live visits
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

However, consider that being cited in AI responses is valuable for brand awareness and traffic, even if your content contributes to training.

Monitoring and Maintenance

Set Up Alerts

Monitor robots.txt access in your web analytics
Set up alerts for unexpected changes to robots.txt
Track AI crawler traffic to verify access

Regular Reviews

Review your robots.txt quarterly
Update when adding new site sections
Remove obsolete rules
Test after any changes

Analytics Integration

Track AI crawler visits:

Set up custom segments in Google Analytics
Filter by user agent for AI crawlers
Monitor pages accessed and patterns
Verify crawlers respect your rules