# Recrawl Frequency Dashboard

The Recrawl Frequency dashboard shows how often crawlers—both search engines, AI platforms, and specialty crawlers—access your site. This helps you understand how quickly changes to your content get picked up and informs your AI visibility report scheduling.

## What You'll Learn

In this guide, you'll learn:

- How to interpret recrawl cadence data
- How to analyze page-level and bot-level patterns
- How to use this data for content strategy
- How to optimize report scheduling

## Prerequisites

This dashboard requires the [Spyglasses AI Traffic Analytics plugin](/docs/getting-started/overview) to be installed on your website. Without the plugin, crawl data cannot be collected.

## Overall Recrawl Cadence

At the top of the dashboard, you'll see your site's average recrawl frequency—how often, on average, crawlers return to your pages.

### What This Metric Means

- **Daily recrawl**: Crawlers visit most pages every day (high activity sites)
- **Weekly recrawl**: Crawlers visit every few days (moderate activity)
- **Monthly recrawl**: Crawlers visit infrequently (low activity or new sites)

Higher recrawl frequency typically indicates:
- Search engines consider your site authoritative
- Content is updated frequently
- Pages are well-linked and discoverable

Lower recrawl frequency may indicate:
- New site still building authority
- Static content that doesn't need frequent updates
- Technical issues preventing efficient crawling

## Page-Level Analysis

Filter to specific pages to see their individual recrawl patterns:

### High-Recrawl Pages
Pages crawled frequently are typically:
- Your homepage
- Category/hub pages
- Recently published content
- Pages with many inbound links

### Low-Recrawl Pages
Pages crawled infrequently may be:
- Deep in site architecture
- Rarely linked to
- Older content
- Pages with technical issues

### Using This Data

If an important page isn't being recrawled:
1. Check that it's in your sitemap
2. Improve internal linking to the page
3. Update the page content to signal freshness
4. Verify no crawl blocks exist

## Bot-Level Analysis

Filter to specific bots to understand their behavior:

### Googlebot
Google's main crawler. High Googlebot activity indicates Google considers your site important. Look for:
- Consistent daily crawls
- Coverage of important pages
- New content discovery timing

### Bingbot
Microsoft's crawler, which powers many AI platforms. Important for AI visibility because Copilot and other AI assistants rely heavily on Bing's index.

### AI Model Training Bots
Bots like GPTBot, ClaudeBot, and Google-Extended crawl content for AI training:
- May visit less frequently than search bots
- Focus on content-rich pages
- Patterns indicate AI interest in your content

### Other Bots
Compare crawl patterns across:
- SEO tool bots (SEMrush, Ahrefs)
- Social platform bots (Facebook, Twitter)
- Other search engines (DuckDuckBot, YandexBot)

## Time-Based Patterns

The timeline chart shows crawl activity over time:

### Identifying Spikes
Sudden increases in crawl activity often indicate:
- New content published
- Site structure changes
- Algorithm updates
- Backlink acquisition

### Identifying Drops
Sudden decreases may indicate:
- Technical issues (robots.txt changes, server problems)
- Content removed or redirected
- Crawl budget changes
- Site penalties (rare)

## Using Recrawl Data for AI Visibility Reports

One key application of this data is **optimizing your AI Visibility Report schedule**.

### Principle
There's no need to run AI visibility reports frequently if search engines haven't recrawled your content. Your visibility won't change if the underlying search index hasn't updated.

### Recommendations

**High recrawl frequency (daily)**
- Run reports weekly to capture changes
- New content will be reflected quickly
- Useful for rapidly changing markets

**Moderate recrawl frequency (weekly)**
- Run reports every 2 weeks
- Changes take time to propagate
- Balance freshness with efficiency

**Low recrawl frequency (monthly)**
- Run reports monthly
- Focus on significant content updates
- Save credits for meaningful comparisons

### Practical Application

1. Check your average recrawl frequency
2. Match your report schedule to this cadence
3. Adjust based on content publication frequency
4. After major updates, wait for recrawl before running reports

## Filters

### Date Range
Select from preset ranges to analyze patterns:
- 7 days for recent activity
- 30 days for monthly patterns
- 90 days for seasonal trends

### Page Path
Filter to specific pages or page patterns to focus analysis.

### Bot Category
Filter by bot type:
- Search crawlers
- AI assistants
- AI trainers
- Specialty bots

### Specific Bot
Drill down to individual bots for detailed analysis.

## Taking Action

### To Increase Recrawl Frequency

1. **Publish fresh content regularly**: Active sites get crawled more
2. **Update existing content**: Signal freshness to crawlers
3. **Improve site architecture**: Better internal linking helps discovery
4. **Submit sitemaps**: Ensure crawlers know about all pages
5. **Build backlinks**: External links encourage crawling

### To Monitor Changes

After making site changes:
1. Note the date of changes
2. Monitor recrawl frequency for affected pages
3. Wait for increased crawl activity
4. Then run AI visibility reports to measure impact

### To Identify Issues

If recrawl frequency drops:
1. Check robots.txt for blocking rules
2. Verify server response times
3. Look for crawl errors in Search Console
4. Ensure pages are accessible

## Best Practices

### Match Content Strategy to Crawl Reality

If your site has low recrawl frequency, don't expect immediate AI visibility changes from content updates. Plan for longer timelines.

### Prioritize High-Impact Pages

Focus optimization efforts on pages that get crawled frequently—changes there will be reflected sooner.

### Use Recrawl Data in Reporting

When presenting AI visibility results to stakeholders, contextualize with recrawl data. If reports show no change after updates, explain that search engines haven't recrawled yet.

### Correlate with Performance

Track correlations between:
- Recrawl frequency and search rankings
- Recrawl frequency and AI mentions
- Content updates and crawl spikes

## Related

- [Bot Traffic](/docs/dashboards/bot-traffic) - See all bot visits in detail
- [AI Visibility Reports](/docs/dashboards/ai-visibility-reports) - Run and schedule reports
- [Settings](/docs/dashboards/settings) - Configure report scheduling
- [Getting Started](/docs/getting-started/overview) - Install the tracking plugin