Cloudflare Workers Integration
Spyglasses integrates seamlessly with Cloudflare Workers to provide edge-based bot detection and AI referrer tracking. This integration is perfect for Webflow sites, e-commerce platforms, and any application that needs to understand and control bot traffic at the edge.
Quick Start with Template
New to Cloudflare Workers? Use our ready-to-deploy template:
🚀 Deploy to Cloudflare Workers
📖 Complete example repository: github.com/orchestra-code/spyglasses-cloudflare-worker-example
✅ No complex configuration required - all setup is done via Cloudflare Workers Routes
Manual Installation
For custom implementations, follow these steps:
-
Install the package:
with yarn:
with pnpm:
-
Create your Worker: Create a new
src/index.ts
file: -
Configure your Worker: Create a
wrangler.toml
file:
Setting Up Workers Routes
Cloudflare Workers Routes determine which requests are sent to your Worker. This is the recommended way to configure your Worker's scope.
Dashboard Configuration
-
Navigate to Workers Routes: In your Cloudflare dashboard, go to your domain and find Workers Routes in the sidebar.
-
Add a Route: Click "Add route" and configure:
- Route pattern:
*your-domain.com/*
(matches all requests to your domain) - Worker: Select your Spyglasses Worker
- Zone: Your domain zone
- Route pattern:
-
Configure Request Limit Failure Mode (Important for high-traffic sites):
- Recommended: Set to "Fail open (proceed)"
- Default: "Fail closed (block)" - ⚠️ Not recommended
- Why this matters: Cloudflare's free plan allows 100,000 Worker requests per day. When this limit is exceeded:
- Fail open: Requests bypass the Worker and go directly to your site (recommended for most users)
- Fail closed: Users receive error pages (can break your site)
📊 High-Traffic Site Considerations
If your site receives more than 100,000 requests per day and you want comprehensive AI traffic tracking, consider:
- Cloudflare Paid Plans: Higher request limits (10M+ requests/month)
- Selective Routing: Use specific route patterns (e.g.,
/blog/*
) to reduce Worker usage - Path Exclusions: Configure
excludePaths
to skip static assets
Wrangler Configuration
You can also configure routes in your wrangler.toml
:
Or using zone ID:
Common Route Patterns:
*example.com/*
- All requests to example.comapi.example.com/*
- Only API subdomainexample.com/blog/*
- Only blog paths*.example.com/*
- All subdomains
For more information, see Cloudflare's Workers Routes documentation.
Configuration
Environment Variables
Variable | Required | Description | Default |
---|---|---|---|
SPYGLASSES_API_KEY | ❌ | API key for analytics and updates | - |
SPYGLASSES_DEBUG | ❌ | Enable debug logging | false |
SPYGLASSES_COLLECTOR_ENDPOINT | ❌ | Custom analytics endpoint | Spyglasses default |
SPYGLASSES_PATTERNS_ENDPOINT | ❌ | Custom endpoint for loading AI and bot patterns | Spyglasses default |
SPYGLASSES_CACHE_TTL | ❌ | Pattern cache TTL in seconds | 3600 |
Setting Your API Key
For analytics and enhanced patterns, set your API key as a secret:
Deployment
Deploy your Worker to Cloudflare:
Advanced Configuration
For custom configuration, create your own worker:
Use Cases
Webflow Sites
Perfect for Webflow sites that need bot protection. Use a route pattern like:
E-commerce Protection
Protect product pages from aggressive scrapers:
API Protection
For API-only protection, use a targeted route:
Subdomain Routing
Route multiple subdomains through the same Worker:
Understanding Detection Results
The worker adds custom headers to help you understand what's happening:
Detection Types
- 🤖 Bot Detection: Identifies crawlers, scrapers, and AI trainers
- 🧠 AI Referrers: Tracks human visitors from AI platforms
- ✅ Legitimate Traffic: Passes through unmodified
Blocking Behavior
- Blocked bots receive a
403 Forbidden
response - AI referrers are logged but never blocked (they're human visitors)
- Regular visitors pass through normally
Debugging
Enable debug mode to see detailed logging:
Check your Worker logs:
You'll see detailed information about:
- Which patterns matched
- Why requests were blocked or allowed
- Logging success/failure
- Performance metrics
Performance
- Sub-millisecond detection using compiled regex patterns
- Minimal memory footprint with smart caching
- Background logging to avoid blocking responses
- Edge distribution for global low latency
- Bundle size: ~66KB total, ~13KB gzipped
Verifying Installation
After deploying your Worker and setting up routes, you can verify the installation by:
- Check logs: Use
npx wrangler tail
to see real-time logs - Test with debug: Enable debug mode to see detailed detection info
- Dashboard data: Check your Spyglasses dashboard for incoming data
- Headers: Look for
X-Spyglasses-Processed: true
in responses
Troubleshooting
Worker Not Triggering
- Verify your route pattern in Cloudflare dashboard under Workers Routes
- Check that the Worker is deployed and the route is correctly configured
- Ensure your domain is using Cloudflare nameservers
Route Pattern Issues
- Use
*domain.com/*
to match all requests to your domain - Use
domain.com/path/*
to match specific paths only - Test route patterns carefully - they're case sensitive for paths
No Data in Dashboard
- Verify your API key is set correctly as a secret
- Check Worker logs for any errors
- Ensure your Workers Route is correctly configured
Performance Issues
- Enable debug mode to identify bottlenecks
- Check your exclude paths configuration
- Verify pattern caching is working (look for cache hit logs)
Security Best Practices
-
Use Secrets: Store your API key as a secret, not a variable
-
Limit Scope: Use specific route patterns to avoid processing unnecessary requests
-
Monitor Logs: Enable debug mode in staging to verify behavior
-
Test Routes: Verify your route patterns match only intended requests
Workers Routes vs Legacy ORIGIN_URL
✅ Workers Routes (Recommended)
- Configured in Cloudflare dashboard or
wrangler.toml
- Cloudflare handles routing automatically
- More flexible pattern matching
- Better performance
❌ ORIGIN_URL (Deprecated)
- Required manual hostname configuration
- Less flexible routing options
- Additional processing overhead
If you're upgrading from a version that used ORIGIN_URL
, simply remove the environment variable and configure Workers Routes instead.
Pricing
Cloudflare Workers usage is included in most Cloudflare plans:
- Free Plan: 100,000 requests/day
- Paid Plans: 10M+ requests/month included
Need help? Contact support@spyglasses.io