Cloudflare Is Blocking Access to Your Site

Learn how Cloudflare's bot protection may be preventing AI assistants from accessing your content.

If you're seeing this page, our crawler detected that Cloudflare is blocking bot access to your site. This affects not just our AI visibility analysis, but also AI assistants like ChatGPT, Claude, and Perplexity.

The Problem: Bot Protection Blocking AI Assistants

Cloudflare's security features are designed to protect your site from malicious bots. However, overly aggressive settings can also block legitimate bots, including:

AI assistant crawlers (GPTBot, ClaudeBot, PerplexityBot)
Our Spyglasses analysis bot
Search engine crawlers
SEO monitoring tools

When these bots try to access your site, Cloudflare returns a challenge page instead of your actual content. AI assistants cannot solve these challenges, which means they cannot access or cite your content.

Why This Matters

When AI assistants are blocked by Cloudflare:

Your content cannot be cited in ChatGPT, Claude, or Perplexity responses
Your brand cannot be recommended when users ask AI for solutions
You're invisible in AI-powered search and recommendations
Competitors who allow AI access gain a significant visibility advantage

In the age of AI-powered search, blocking AI crawlers is like blocking Google—you're removing yourself from a major discovery channel.

How to Check Your Settings

1. Verify Bot Protection Level

Log into your Cloudflare dashboard
Select your domain
Go to Security → Bots
Check your Bot Fight Mode setting

High or aggressive settings will block AI assistants.

2. Check WAF Rules

Go to Security → WAF
Review your Custom Rules and Rate Limiting Rules
Look for rules that might be blocking crawlers

Common problematic rules:

Blocking all non-browser user agents
Very low rate limits (< 10 requests per minute)
Challenge or block on "Known Bots"

3. Review Firewall Rules

Go to Security → Firewall Rules
Look for rules that challenge or block based on:
- User Agent patterns
- ASN (Autonomous System Numbers)
- Country blocking

How to Fix It

Option 1: Allow Verified AI Crawlers (Recommended)

Create a Custom Rule to allow verified AI crawlers:

Go to Security → WAF → Custom Rules
Click Create Rule
Set up the rule:

Rule name: Allow AI Assistants
Field: User Agent
Operator: contains
Value: GPTBot OR ClaudeBot OR PerplexityBot OR anthropic-ai OR Spyglasses
Action: Skip → All remaining custom rules

Save and deploy

This allows AI assistants while keeping your other security measures active.

Option 2: Adjust Bot Fight Mode

If you have Bot Fight Mode enabled:

Go to Security → Bots
Change from "Fight" to "Allow Verified Bots"
Save changes

This is less precise but easier to configure.

Option 3: Create Verified Bot Allowlist

For more control:

Go to Security → WAF → Custom Rules
Create a rule for verified bots:

Rule name: Verified Bot Allowlist
Expression: 
  cf.client.bot OR 
  cf.bot_management.verified_bot
Action: Allow

This uses Cloudflare's own bot verification system.

Recommended AI Crawler User Agents

Here are the user agents you should allow:

AI Assistant	User Agent Pattern
ChatGPT	`GPTBot`
Claude	`ClaudeBot` or `anthropic-ai`
Perplexity	`PerplexityBot`
Google Gemini	`Google-Extended`
Microsoft Copilot	`Bingbot`
Spyglasses	`Spyglasses`

Full Allowlist Example

(http.user_agent contains "GPTBot") or 
(http.user_agent contains "ClaudeBot") or 
(http.user_agent contains "anthropic-ai") or 
(http.user_agent contains "PerplexityBot") or 
(http.user_agent contains "Google-Extended") or 
(http.user_agent contains "Bingbot") or
(http.user_agent contains "Spyglasses")