Bot Traffic Dashboard

The Bot Traffic dashboard shows all visits to your site from the over 600 bots that Spyglasses tracks. This includes AI assistants, AI model training crawlers, traditional search engine crawlers, specialty bots, and scrapers.

What You'll Learn

In this guide, you'll learn:

• What types of bots visit your site
• How to filter and analyze bot traffic
• How to identify potentially harmful bot activity
• How to use this data for optimization

Prerequisites

This dashboard requires the Spyglasses AI Traffic Analytics plugin to be installed on your website. Without the plugin, no data will appear.

Bot Categories

Spyglasses categorizes bots into several types:

AI Assistants

Bots that power AI chat interfaces:

• ChatGPT
• Claude
• Google Gemini
• Perplexity
• Microsoft Copilot
• And many others

These visits represent AI platforms actively using your content to answer user questions.

AI Model Trainers

Crawlers that collect data for training AI models:

• GPTBot (OpenAI)
• ClaudeBot (Anthropic)
• Google-Extended
• Meta-ExternalAgent
• CCBot (Common Crawl)

These bots are gathering your content for future AI model training, not current conversations.

Search Engine Crawlers

Traditional search engine indexing bots:

• Googlebot
• Bingbot
• DuckDuckBot
• YandexBot
• Baidu Spider

These crawlers index your content for traditional search results.

Specialty Bots

Specific-purpose crawlers:

• SEMrush Bot
• Ahrefs Bot
• Majestic Bot
• Facebook External Hit
• Twitter Bot
• LinkedIn Bot

These crawlers serve specific tools and platforms.

Scrapers and Others

General-purpose tools:

• cURL
• wget
• Python requests
• Various scraping tools

Some of these are legitimate (developers, testing), others may be extracting content without permission.

Filtering Options

By Category

Filter to specific bot categories to focus your analysis:

• Select "AI Assistants" to see only AI chat traffic
• Select "AI Model Trainers" to monitor training crawlers
• Select "Search Crawlers" to focus on indexing activity

By Specific Bot

Drill down to individual bots:

• See exactly which version of Googlebot visited
• Monitor specific AI assistants
• Track individual specialty bots

By Page

Filter to specific pages to see which bots access them:

• Identify pages attracting AI attention
• See what search engines are crawling
• Detect pages being scraped

By Location

Filter by country, region, or city to understand geographic patterns:

• Identify traffic from unexpected locations
• Detect potentially suspicious regional patterns
• Understand where your bot traffic originates

By Intent

Filter by page intent classification:

• See which content types attract which bots
• Understand AI vs. search crawler preferences
• Identify gaps in bot coverage

Learn more at Page Intent Classification.

Traffic Timeline

The main chart shows bot traffic over time with breakdown by category. Use this to:

• Spot unusual spikes in bot activity
• Identify crawl patterns
• Monitor for potential attacks or abuse

Bot Details Table

The detailed table shows:

Click any row to see detailed visit logs for that bot.

Using This Data

Monitor AI Coverage

High AI assistant traffic indicates your content is being actively used in AI conversations. Low traffic might mean:

• Content isn't being discovered
• Content isn't considered authoritative
• Technical issues preventing access

Verify Search Engine Access

Ensure search engines can crawl your important pages:

• Check that Googlebot and Bingbot visit key pages
• Monitor crawl frequency for important content
• Identify pages being ignored

Detect Suspicious Activity

Watch for warning signs:

• Sudden spikes from unknown bots
• High-volume scraping from suspicious sources
• Unusual geographic patterns
• Bots accessing sensitive pages

Optimize Based on Crawler Behavior

Understanding what bots access helps optimization:

• If AI prefers certain content, create more
• If crawlers skip pages, check technical issues
• If scrapers target specific content, consider protection

Complete Bot List

For a complete list of all bots Spyglasses tracks, visit /bots. This includes:

• Bot name and company
• Category classification
• Description and purpose
• User agent strings

Traffic Control Integration

If you see unwanted bot traffic, you can block specific bots or categories using Traffic Control:

1. Identify the bot or category to block
2. Go to Traffic Control
3. Add a block rule
4. Monitor to confirm blocking works

Best Practices

Review Weekly

Check bot traffic at least weekly to:

• Spot trends early
• Identify new bots
• Detect potential issues

Separate AI Assistants from AI Trainers

These serve different purposes:

• AI Assistants use content now (valuable for visibility)
• AI Trainers collect for future use (consider your data policy)

Don't Block Everything

Blocking too aggressively can hurt you:

• Search crawlers need access for SEO
• AI assistants need access for visibility
• Some specialty bots drive referral traffic

Cross-Reference with Analytics

Compare bot traffic to your main analytics:

• Do crawl patterns align with indexing?
• Does AI assistant traffic correlate with AI-referred visits?
• Are there discrepancies indicating issues?

Related

• AI Traffic Analytics - Focus on AI assistant traffic
• Traffic Control - Block or allow specific bots
• Recrawl Frequency - Understand crawl patterns
• Bot Directory - Full list of tracked bots
• Getting Started - Install the tracking plugin