Traffic Control

Spyglasses gives you powerful control over which AI agents and bots can access your website. Traffic Control is managed centrally in the Spyglasses app at the property level. Your site integration (middleware, plugin, or worker) automatically syncs the canonical rules and enforces them at the edge, so you don’t have to ship code changes to update block/allow lists.

What You'll Learn

In this guide, you'll learn how to:

• Configure basic bot blocking settings
• Create custom block and allow rules
• Exclude specific paths from monitoring
• Implement traffic control on different platforms
• Use advanced pattern matching for fine-grained control

Centralized Management

All bot rules are configured in the Spyglasses app and applied per property. Your integrations periodically sync the latest pattern library and your block/allow rules and then enforce them automatically.

Global Settings

Enable broad protections with a single switch. For example, turn on Block AI Model Trainers to automatically block known model-training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, etc.).

Custom Rules

Create fine‑grained rules by category or specific pattern. You can browse categories, view their patterns, and set Block or Allow for each.

Browse categories and patterns and set Block/Allow per item. Changes are saved to your property and synced to all integrations.

Rules Summary

Review the effective set of rules currently applied to your property.

A concise view of which categories and patterns are blocked or allowed. This is exactly what your site will enforce.

Optional: Path Exclusions (in code)

You can optionally exclude certain paths from Spyglasses monitoring and enforcement in your code. This is useful for health checks, admin pages, or internal routes that should bypass Traffic Control.

How It Works With Your Site

Your plugin/middleware enforces the centralized rules—no code changes are needed to add or update block/allow rules. For Next.js applications, keep your middleware minimal and let Spyglasses handle rule evaluation.

// middleware.ts
import { createSpyglassesMiddleware } from '@spyglasses/next';
 
export default createSpyglassesMiddleware({
  apiKey: process.env.SPYGLASSES_API_KEY,
  debug: process.env.SPYGLASSES_DEBUG === 'true',
  // Optional: exclude paths from Spyglasses monitoring/enforcement
  excludePaths: ['/health', '/api/status', /^\/admin/, /^\/internal/],
});
 
export const config = {
  matcher: ['/((?!_next|api|favicon.ico|.*\\.(jpg|jpeg|gif|png|svg|ico|css|js)).*)'],
};

Manage Everything in the Spyglasses App

Whether you use Next.js, WordPress, Cloudflare Workers, Rails, or Django, you manage Traffic Control in a single place—the Spyglasses app. Integrations simply enforce whatever is configured for your property.

Example Rule Sets (configure in the app)

Protecting specific content

• Block: category AI Model Trainers, category Scraper
• Allow: Googlebot, Bingbot, facebookexternalhit, TwitterBot
• Optional: exclude premium/member paths in code if you do not want them monitored

E‑commerce protection

• Block: category Scraper, pattern
  • .*price.* (price monitors)
• Allow: Googlebot, ShoppingBot

Content publisher setup

• Block: category AI, category Scraper
• Allow: Googlebot, Bingbot, facebookexternalhit, TwitterBot
• Optional exclusions in code: /sitemap.xml, /robots.txt, and paths matching /^\/feed/

Testing Your Configuration

After implementing traffic control, you can test your configuration:

1. Check the Spyglasses dashboard to see which bots are being blocked
2. Monitor your server logs for blocked requests
3. Use browser developer tools to test excluded paths
4. Verify search engine access using Google Search Console

Best Practices

Start Conservative

Begin with basic settings and gradually add more restrictive rules:

Start with the Global Settings toggle for Block AI Model Trainers, then refine using category/pattern rules. No code changes are required to adjust rules—use the Spyglasses app.

Always allow search engines

Ensure legitimate search engines can access your content by setting Allow rules in the Spyglasses app for: Googlebot, Bingbot, DuckDuckBot, YandexBot.

Monitor Impact

Regularly check your analytics to ensure you're not blocking legitimate traffic. The Spyglasses dashboard provides detailed reports on blocked requests.

Use exclusions wisely

If needed, exclude paths that do not require protection or monitoring in your middleware configuration (e.g., /health, /api/public, /^\/static\//).

Troubleshooting

Bot Still Getting Through

If unwanted bots are still accessing your site:

1. Check if they match an allow rule
2. Verify your patterns are correct
3. Look for new bot user agents in your logs
4. Contact support for help with custom patterns

Legitimate Traffic Blocked

If you're accidentally blocking legitimate traffic:

1. Add specific allow rules for important bots
2. Check your custom block patterns aren't too broad
3. Review your exclusion paths
4. Test with debug mode enabled