# Bot Traffic Dashboard

The Bot Traffic dashboard shows all visits to your site from the over [600 bots](/bots) that Spyglasses tracks. This includes AI assistants, AI model training crawlers, traditional search engine crawlers, specialty bots, and scrapers.

## What You'll Learn

In this guide, you'll learn:

- What types of bots visit your site
- How to filter and analyze bot traffic
- How to identify potentially harmful bot activity
- How to use this data for optimization

## Prerequisites

This dashboard requires the [Spyglasses AI Traffic Analytics plugin](/docs/getting-started/overview) to be installed on your website. Without the plugin, no data will appear.

## Bot Categories

Spyglasses categorizes bots into several types:

### AI Assistants
Bots that power AI chat interfaces:
- ChatGPT
- Claude
- Google Gemini
- Perplexity
- Microsoft Copilot
- And many others

These visits represent AI platforms actively using your content to answer user questions.

### AI Model Trainers
Crawlers that collect data for training AI models:
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended
- Meta-ExternalAgent
- CCBot (Common Crawl)

These bots are gathering your content for future AI model training, not current conversations.

### Search Engine Crawlers
Traditional search engine indexing bots:
- Googlebot
- Bingbot
- DuckDuckBot
- YandexBot
- Baidu Spider

These crawlers index your content for traditional search results.

### Specialty Bots
Specific-purpose crawlers:
- SEMrush Bot
- Ahrefs Bot
- Majestic Bot
- Facebook External Hit
- Twitter Bot
- LinkedIn Bot

These crawlers serve specific tools and platforms.

### Scrapers and Others
General-purpose tools:
- cURL
- wget
- Python requests
- Various scraping tools

Some of these are legitimate (developers, testing), others may be extracting content without permission.

## Filtering Options

### By Category

Filter to specific bot categories to focus your analysis:
- Select "AI Assistants" to see only AI chat traffic
- Select "AI Model Trainers" to monitor training crawlers
- Select "Search Crawlers" to focus on indexing activity

### By Specific Bot

Drill down to individual bots:
- See exactly which version of Googlebot visited
- Monitor specific AI assistants
- Track individual specialty bots

### By Page

Filter to specific pages to see which bots access them:
- Identify pages attracting AI attention
- See what search engines are crawling
- Detect pages being scraped

### By Location

Filter by country, region, or city to understand geographic patterns:
- Identify traffic from unexpected locations
- Detect potentially suspicious regional patterns
- Understand where your bot traffic originates

### By Intent

Filter by page intent classification:
- See which content types attract which bots
- Understand AI vs. search crawler preferences
- Identify gaps in bot coverage

Learn more at [Page Intent Classification](/docs/getting-started/page-intent-classification).

## Traffic Timeline

The main chart shows bot traffic over time with breakdown by category. Use this to:
- Spot unusual spikes in bot activity
- Identify crawl patterns
- Monitor for potential attacks or abuse

## Bot Details Table

The detailed table shows:

| Column | Description |
|--------|-------------|
| **Bot Name** | The identified bot |
| **Category** | Bot category (AI, Search, etc.) |
| **Visits** | Total visits in selected period |
| **Pages** | Unique pages visited |
| **Last Seen** | Most recent visit |

Click any row to see detailed visit logs for that bot.

## Using This Data

### Monitor AI Coverage

High AI assistant traffic indicates your content is being actively used in AI conversations. Low traffic might mean:
- Content isn't being discovered
- Content isn't considered authoritative
- Technical issues preventing access

### Verify Search Engine Access

Ensure search engines can crawl your important pages:
- Check that Googlebot and Bingbot visit key pages
- Monitor crawl frequency for important content
- Identify pages being ignored

### Detect Suspicious Activity

Watch for warning signs:
- Sudden spikes from unknown bots
- High-volume scraping from suspicious sources
- Unusual geographic patterns
- Bots accessing sensitive pages

### Optimize Based on Crawler Behavior

Understanding what bots access helps optimization:
- If AI prefers certain content, create more
- If crawlers skip pages, check technical issues
- If scrapers target specific content, consider protection

## Complete Bot List

For a complete list of all bots Spyglasses tracks, visit [/bots](/bots). This includes:
- Bot name and company
- Category classification
- Description and purpose
- User agent strings

## Traffic Control Integration

If you see unwanted bot traffic, you can block specific bots or categories using [Traffic Control](/docs/dashboards/traffic-control):
1. Identify the bot or category to block
2. Go to Traffic Control
3. Add a block rule
4. Monitor to confirm blocking works

## Best Practices

### Review Weekly

Check bot traffic at least weekly to:
- Spot trends early
- Identify new bots
- Detect potential issues

### Separate AI Assistants from AI Trainers

These serve different purposes:
- AI Assistants use content now (valuable for visibility)
- AI Trainers collect for future use (consider your data policy)

### Don't Block Everything

Blocking too aggressively can hurt you:
- Search crawlers need access for SEO
- AI assistants need access for visibility
- Some specialty bots drive referral traffic

### Cross-Reference with Analytics

Compare bot traffic to your main analytics:
- Do crawl patterns align with indexing?
- Does AI assistant traffic correlate with AI-referred visits?
- Are there discrepancies indicating issues?

## Related

- [AI Traffic Analytics](/docs/dashboards/ai-traffic-analytics) - Focus on AI assistant traffic
- [Traffic Control](/docs/dashboards/traffic-control) - Block or allow specific bots
- [Recrawl Frequency](/docs/dashboards/recrawl-frequency) - Understand crawl patterns
- [Bot Directory](/bots) - Full list of tracked bots
- [Getting Started](/docs/getting-started/overview) - Install the tracking plugin