Cloudflare Workers Integration

Spyglasses integrates seamlessly with Cloudflare Workers to provide edge-based bot detection and AI referrer tracking. This integration is perfect for Webflow sites, e-commerce platforms, and any application that needs to understand and control bot traffic at the edge.

Quick Start with Template

New to Cloudflare Workers? Use our ready-to-deploy template:

📖 Complete example repository: github.com/orchestra-code/spyglasses-cloudflare-worker-example

✅ No complex configuration required - all setup is done via Cloudflare Workers Routes

Manual Installation

For custom implementations, follow these steps:

1. Install the package:

   npm install @spyglasses/cloudflare-worker

   with yarn:

   yarn add @spyglasses/cloudflare-worker

   with pnpm:

   pnpm add @spyglasses/cloudflare-worker

2. Create your Worker: Create a new src/index.ts file:

   import worker from '@spyglasses/cloudflare-worker';
    
   export default worker;

3. Configure your Worker: Create a wrangler.toml file:

   name = "my-spyglasses-worker"
   main = "src/index.ts"
   compatibility_date = "2024-12-30"
   compatibility_flags = ["nodejs_compat"]
    
   [vars]
   SPYGLASSES_DEBUG = "false"

Setting Up Workers Routes

Cloudflare Workers Routes determine which requests are sent to your Worker. This is the recommended way to configure your Worker's scope.

Dashboard Configuration

1. Navigate to Workers Routes: In your Cloudflare dashboard, go to your domain and find Workers Routes in the sidebar.

2. Add a Route: Click "Add route" and configure:

   • Route pattern: *your-domain.com/* (matches all requests to your domain)
   • Worker: Select your Spyglasses Worker
   • Zone: Your domain zone

3. Configure Request Limit Failure Mode (Important for high-traffic sites):

   • Recommended: Set to "Fail open (proceed)"
   • Default: "Fail closed (block)" - ⚠️ Not recommended
   • Why this matters: Cloudflare's free plan allows 100,000 Worker requests per day. When this limit is exceeded:
     • Fail open: Requests bypass the Worker and go directly to your site (recommended for most users)
     • Fail closed: Users receive error pages (can break your site)

📊 High-Traffic Site Considerations

If your site receives more than 100,000 requests per day and you want comprehensive AI traffic tracking, consider:

• Cloudflare Paid Plans: Higher request limits (10M+ requests/month)
• Selective Routing: Use specific route patterns (e.g., /blog/*) to reduce Worker usage
• Path Exclusions: Configure excludePaths to skip static assets

Wrangler Configuration

You can also configure routes in your wrangler.toml:

[[routes]]
pattern = "*your-domain.com/*"
zone_name = "your-domain.com"

Or using zone ID:

[[routes]]
pattern = "*your-domain.com/*"
zone_id = "your-zone-id-here"

Common Route Patterns:

• *example.com/* - All requests to example.com
• api.example.com/* - Only API subdomain
• example.com/blog/* - Only blog paths
• *.example.com/* - All subdomains

For more information, see Cloudflare's Workers Routes documentation.

Configuration

Environment Variables

Setting Your API Key

For analytics and enhanced patterns, set your API key as a secret:

npx wrangler secret put SPYGLASSES_API_KEY
# Enter your API key when prompted

Deployment

Deploy your Worker to Cloudflare:

npx wrangler deploy

Advanced Configuration

For custom configuration, create your own worker:

import { createSpyglassesWorker, SpyglassesWorkerConfig } from '@spyglasses/cloudflare-worker';
 
const config: SpyglassesWorkerConfig = {
  debug: true,
  excludePaths: [
    '/api/',
    '/admin/',
    '/_next/',
    /\.(css|js|png|jpg|gif|svg|ico)$/i, // Regex patterns supported
  ],
  platformType: 'my-platform',
  blockingTimeout: 1000, // Max time to wait for logging blocked requests
  awaitBlockedLogging: false, // Don't block response on logging
};
 
export default {
  async fetch(request: Request, env: any, ctx: ExecutionContext): Promise<Response> {
    const spyglassesWorker = createSpyglassesWorker({
      ...config,
      apiKey: env.SPYGLASSES_API_KEY,
      debug: env.SPYGLASSES_DEBUG === 'true',
    });
 
    return await spyglassesWorker.handleRequest(request, env, ctx);
  },
};

Use Cases

Webflow Sites

Perfect for Webflow sites that need bot protection. Use a route pattern like:

[[routes]]
pattern = "*your-site.webflow.io/*"
zone_name = "webflow.io"

E-commerce Protection

Protect product pages from aggressive scrapers:

const config: SpyglassesWorkerConfig = {
  excludePaths: [
    '/cart',
    '/checkout',
    '/account',
    '/admin',
  ],
  // Block AI model trainers but allow price comparison bots
};

API Protection

For API-only protection, use a targeted route:

[[routes]]
pattern = "api.example.com/*"
zone_name = "example.com"

Subdomain Routing

Route multiple subdomains through the same Worker:

[[routes]]
pattern = "*.example.com/*"
zone_name = "example.com"

Understanding Detection Results

The worker adds custom headers to help you understand what's happening:

X-Spyglasses-Processed: true
X-Spyglasses-Blocked: true (if blocked)
X-Spyglasses-Reason: bot|ai_referrer (if blocked)

Detection Types

• 🤖 Bot Detection: Identifies crawlers, scrapers, and AI trainers
• 🧠 AI Referrers: Tracks human visitors from AI platforms
• ✅ Legitimate Traffic: Passes through unmodified

Blocking Behavior

• Blocked bots receive a 403 Forbidden response
• AI referrers are logged but never blocked (they're human visitors)
• Regular visitors pass through normally

Debugging

Enable debug mode to see detailed logging:

[vars]
SPYGLASSES_DEBUG = "true"

Check your Worker logs:

npx wrangler tail

You'll see detailed information about:

• Which patterns matched
• Why requests were blocked or allowed
• Logging success/failure
• Performance metrics

Performance

• Sub-millisecond detection using compiled regex patterns
• Minimal memory footprint with smart caching
• Background logging to avoid blocking responses
• Edge distribution for global low latency
• Bundle size: ~66KB total, ~13KB gzipped

Verifying Installation

After deploying your Worker and setting up routes, you can verify the installation by:

1. Check logs: Use npx wrangler tail to see real-time logs
2. Test with debug: Enable debug mode to see detailed detection info
3. Dashboard data: Check your Spyglasses dashboard for incoming data
4. Headers: Look for X-Spyglasses-Processed: true in responses

Troubleshooting

Worker Not Triggering

• Verify your route pattern in Cloudflare dashboard under Workers Routes
• Check that the Worker is deployed and the route is correctly configured
• Ensure your domain is using Cloudflare nameservers

Route Pattern Issues

• Use *domain.com/* to match all requests to your domain
• Use domain.com/path/* to match specific paths only
• Test route patterns carefully - they're case sensitive for paths

No Data in Dashboard

• Verify your API key is set correctly as a secret
• Check Worker logs for any errors
• Ensure your Workers Route is correctly configured

Performance Issues

• Enable debug mode to identify bottlenecks
• Check your exclude paths configuration
• Verify pattern caching is working (look for cache hit logs)

Security Best Practices

1. Use Secrets: Store your API key as a secret, not a variable

   npx wrangler secret put SPYGLASSES_API_KEY

2. Limit Scope: Use specific route patterns to avoid processing unnecessary requests

3. Monitor Logs: Enable debug mode in staging to verify behavior

4. Test Routes: Verify your route patterns match only intended requests

Cost Implications for using Cloudflare

Cloudflare Workers usage is included in most Cloudflare plans:

• Free Plan: 100,000 requests/day (set your worker to Fail Open to avoid overage)
• Paid Plans: 10M+ requests/month included

Need help? Contact support@spyglasses.io