Documentation
Dashboard Guides/Traffic Control Dashboard

Traffic Control Dashboard

The Traffic Control dashboard lets you configure which AI agents and bots can access your website. Rules are managed centrally in Spyglasses and automatically enforced by your site integration.

For comprehensive Traffic Control documentation including implementation details and example configurations, see Traffic Control Guide.

What You'll Learn

In this guide, you'll learn:

  • How to configure bot blocking settings
  • How to create custom block and allow rules
  • How rules sync to your site
  • Best practices for bot management

How It Works

Traffic Control is managed centrally in Spyglasses at the property level. Your site integration (middleware, plugin, or worker) automatically syncs the rules and enforces them at the edge. You don't need to ship code changes to update your block/allow lists.

Rule Enforcement

  1. You configure rules in Spyglasses
  2. Your integration periodically syncs rules
  3. When bots visit, rules are applied automatically
  4. Blocked bots receive HTTP 403 Forbidden responses
  5. Allowed bots access your content normally

Global Settings

Enable broad protections with a single toggle:

Block AI Model Trainers

When enabled, this automatically blocks known model-training crawlers:

  • GPTBot (OpenAI)
  • ClaudeBot (Anthropic)
  • Meta-ExternalAgent
  • Google-Extended
  • CCBot (Common Crawl)
  • And others

This prevents your content from being used in future AI training while still allowing AI assistants to access your content for current conversations.

Other Global Settings

Depending on your configuration options:

  • Block all scrapers
  • Block suspicious user agents

Browse Categories

View bot categories and their associated patterns:

  • AI Model Trainers
  • AI Assistants
  • Search Crawlers
  • SEO Bots
  • Social Bots
  • Scrapers

For each category, you can set Block or Allow.

Browse Specific Patterns

Drill down to specific bots or user agent patterns:

  • Set rules for individual bots
  • Create pattern-based rules
  • Override category-level settings

Rule Priority

More specific rules override broader rules:

  1. Specific bot rules (highest priority)
  2. Pattern-based rules
  3. Category rules
  4. Global settings (lowest priority)

Rules Summary

The Rules Summary view shows exactly what's being enforced:

  • All blocked categories and patterns
  • All allowed overrides
  • Effective rule set

This is what your site actually enforces—review it to ensure your configuration is correct.

Creating Rules

To Block a Bot or Category

  1. Find the bot or category in the browse view
  2. Click to select it
  3. Choose "Block"
  4. Save changes

To Allow a Blocked Bot

If a bot is blocked by category but you want to allow it:

  1. Find the specific bot
  2. Choose "Allow"
  3. Save changes

The allow rule overrides the category block.

Pattern-Based Rules

For advanced filtering:

  1. Create a custom pattern
  2. Enter a user agent substring or regex
  3. Set Block or Allow
  4. Save changes

Path Exclusions

You can optionally exclude certain paths from Traffic Control in your code. This is useful for:

  • Health check endpoints
  • Admin pages
  • Internal APIs
  • Public assets

See Traffic Control Guide for implementation details.

Testing Your Configuration

After configuring rules:

  1. Check Spyglasses dashboard: Review blocked vs. allowed traffic
  2. Monitor server logs: Verify blocked requests are handled correctly
  3. Test important bots: Ensure Googlebot and other critical bots can access
  4. Verify search access: Check Google Search Console for crawl issues

Best Practices

Start Conservative

Begin with minimal blocking:

  1. Enable "Block AI Model Trainers" if desired
  2. Monitor traffic for a week
  3. Add additional rules based on what you observe

Always Allow Search Engines

Ensure legitimate search engines can access your content:

  • Googlebot (required for Google Search)
  • Bingbot
  • DuckDuckBot
  • YandexBot (if targeting that market)

Blocking search engines harms your visibility.

Don't Block AI Assistants

If you want AI visibility, don't block AI assistant bots. Block only training crawlers if you want to prevent future AI training while maintaining current AI visibility.

Review Regularly

Periodically review:

  • What's being blocked
  • Any new bots that appeared
  • Whether rules are still appropriate

Monitor Impact

Watch your analytics after making changes:

  • Did bot traffic decrease as expected?
  • Are there unintended side effects?
  • Is legitimate traffic affected?

Troubleshooting

Bot Still Getting Through

If blocked bots still access your site:

  1. Check if they match an allow rule
  2. Verify your patterns are correct
  3. Ensure your integration synced rules

Legitimate Traffic Blocked

If you're accidentally blocking legitimate traffic:

  1. Add specific allow rules for important bots
  2. Review custom patterns for overly broad matches
  3. Check category blocks for unintended coverage

Rules Not Syncing

If changes aren't taking effect:

  1. Verify your integration is running
  2. Check for sync errors in logs
  3. Force a manual sync if available
  4. Verify API connection

Previous

Recrawl Frequency Dashboard

Next

Property Pages Dashboard

On this page