Recrawl Frequency Dashboard

The Recrawl Frequency dashboard shows how often crawlers—both search engines, AI platforms, and specialty crawlers—access your site. This helps you understand how quickly changes to your content get picked up and informs your AI visibility report scheduling.

What You'll Learn

In this guide, you'll learn:

How to interpret recrawl cadence data
How to analyze page-level and bot-level patterns
How to use this data for content strategy
How to optimize report scheduling

Prerequisites

This dashboard requires the Spyglasses AI Traffic Analytics plugin to be installed on your website. Without the plugin, crawl data cannot be collected.

Overall Recrawl Cadence

At the top of the dashboard, you'll see your site's average recrawl frequency—how often, on average, crawlers return to your pages.

What This Metric Means

Daily recrawl: Crawlers visit most pages every day (high activity sites)
Weekly recrawl: Crawlers visit every few days (moderate activity)
Monthly recrawl: Crawlers visit infrequently (low activity or new sites)

Higher recrawl frequency typically indicates:

Search engines consider your site authoritative
Content is updated frequently
Pages are well-linked and discoverable

Lower recrawl frequency may indicate:

New site still building authority
Static content that doesn't need frequent updates
Technical issues preventing efficient crawling

Page-Level Analysis

Filter to specific pages to see their individual recrawl patterns:

High-Recrawl Pages

Pages crawled frequently are typically:

Your homepage
Category/hub pages
Recently published content
Pages with many inbound links

Low-Recrawl Pages

Pages crawled infrequently may be:

Deep in site architecture
Rarely linked to
Older content
Pages with technical issues

Using This Data

If an important page isn't being recrawled:

Check that it's in your sitemap
Improve internal linking to the page
Update the page content to signal freshness
Verify no crawl blocks exist

Bot-Level Analysis

Filter to specific bots to understand their behavior:

Googlebot

Google's main crawler. High Googlebot activity indicates Google considers your site important. Look for:

Consistent daily crawls
Coverage of important pages
New content discovery timing

Bingbot

Microsoft's crawler, which powers many AI platforms. Important for AI visibility because Copilot and other AI assistants rely heavily on Bing's index.

AI Model Training Bots

Bots like GPTBot, ClaudeBot, and Google-Extended crawl content for AI training:

May visit less frequently than search bots
Focus on content-rich pages
Patterns indicate AI interest in your content

Other Bots

Compare crawl patterns across:

SEO tool bots (SEMrush, Ahrefs)
Social platform bots (Facebook, Twitter)
Other search engines (DuckDuckBot, YandexBot)

Time-Based Patterns

The timeline chart shows crawl activity over time:

Identifying Spikes

Sudden increases in crawl activity often indicate:

New content published
Site structure changes
Algorithm updates
Backlink acquisition

Identifying Drops

Sudden decreases may indicate:

Technical issues (robots.txt changes, server problems)
Content removed or redirected
Crawl budget changes
Site penalties (rare)

Using Recrawl Data for AI Visibility Reports

One key application of this data is optimizing your AI Visibility Report schedule.

Principle

There's no need to run AI visibility reports frequently if search engines haven't recrawled your content. Your visibility won't change if the underlying search index hasn't updated.