How to Time Content Updates to AI Recrawl Cadence

To time content updates to AI recrawl cadence, start by measuring how often crawlers actually return to your pages, using server-side logs rather than a guess. Break that number down per page to find important URLs that go weeks between visits, then per bot so you know which of those visits feed the search indexes AI assistants query. Publish and update against the interval you measured, not a calendar someone picked in a planning meeting, and schedule visibility reports at least one full interval after the change plus a week. Then confirm the page was refetched before you decide whether the update worked. Most sites find their average recrawl is far slower than their publishing schedule, which explains a lot of updates that seemed to do nothing.

Here's the version of this that shows up in a marketing meeting. You updated eleven pages last quarter. Somebody ran a visibility check two weeks later, and nothing moved. So the team concludes the content work isn't paying off, and the next quarter's plan gets smaller.

There's a decent chance nothing was wrong with the content. The pages just hadn't been read yet.

Why the delay exists

An update to an existing page usually takes two to six weeks to show up in AI answers. On sites crawled daily it can be under a week; on sites crawled monthly, eight weeks is normal. The delay isn't the assistant being slow. It's that the assistant never sees your page until a crawler brings it in.

AI assistants don't visit your site when someone asks a question about your category. For most questions, they run web searches, get back results from an index, and read pages that were already fetched and stored. The version of your page in that pipeline is whatever a crawler last picked up.

So there's a queue between you and the answer, and it has stages:

1. A crawler such as Googlebot or Bingbot returns to your page and re-reads it. This is the long pole, and on most B2B sites it takes 2 to 5 weeks.
2. The index gets updated with the new version, usually within a few days of that visit.
3. An assistant runs a search that surfaces your page.
4. The assistant reads it and builds an answer.

Nothing downstream happens until step one happens. On a site crawled every five weeks, a page you updated on the first of the month might not be visible as updated until well into the next one. Meanwhile your dashboard shows the old numbers and everyone reads that as failure.

This guide works with any AI visibility setup; the callouts show how each step runs in Spyglasses.

Step	What you're doing	What tells you it's working
1. Measure the cadence	Getting a real site-wide interval	A number in days, not a feeling
2. Find starved pages	Breaking cadence down per URL	A short list of neglected important pages
3. Split by bot	Seeing who visits and how often	You know which bots gate your answers
4. Time the calendar	Matching work and reports to the interval	Reports land after pickup, not before
5. Verify pickup	Confirming the refetch happened	A crawl date after your publish date

Measure how often crawlers actually come back

Start with one number for the whole site. How many days pass, on average, between crawler visits to the same page?

You need server-side data to answer that. Every crawler that reads your site makes a request to your server and identifies itself in the user agent, so a log of those requests gives you the true interval. Analytics that runs in the browser won't see any of it, because bots don't execute your JavaScript. Search Console gets you a partial view of one company's crawler and says nothing about the rest.

When teams see the real number for the first time, the reaction is usually surprise in one direction. Either the site is crawled far less than assumed, which reframes months of disappointing content results, or a handful of pages are crawled constantly while everything else sits untouched, which is its own problem.

A rough read on what you'll find:

• Daily. Big, active, well-linked sites. News, large ecommerce, sites publishing several times a week.
• Weekly. Most established B2B sites with a regular blog. This is the common case.
• Monthly or slower. Newer sites, small sites, or sites that publish rarely. Also sites with technical problems that make crawling expensive.

None of these is a grade. A slow cadence on a stable site isn't a failure, it's information you need before you plan anything. The mistake is planning a six-week campaign as though you're on a daily cadence when you're on a monthly one.

Find the pages AI rarely refetches

The site-wide average hides the thing you need. Crawl attention isn't spread evenly, and it's usually concentrated on pages that don't need it.

Your homepage gets read constantly. So do your main category pages and whatever you published last week. Meanwhile the pages that answer real buying questions, the comparison page, the pricing explainer, the product documentation, can go a month or more between visits. Those are the starved pages, and they're often the exact ones you'd want to update for AI visibility.

Sort your important URLs by days since last crawl and look at the top of that list. You're looking for the gap between commercial importance and crawl attention. A page nobody visits and nobody cares about being crawled twice a year is fine. A pricing page in the same state is a problem.

Starvation usually has one of four causes, and they're worth distinguishing because the fixes differ:

Cause	What it looks like	Fix
Buried in the architecture	Four or more clicks from the homepage	Link it from a page crawled weekly
Missing from the sitemap	Not listed, or a wrong lastmod date	Fix the sitemap entry
Never updated	Last change 18 months ago	Give it a real update, not a date bump
Technically expensive	Slow response, redirect chain, heavy render	Fix the response before anything else

The first one is the most common and the easiest win. Crawlers find pages by following links, so a page linked only from a footer or a deep archive gets visited on that schedule. Adding a contextual link from a page that gets crawled every few days usually pulls the starved page up within a couple of weeks. It costs one edit.

Skip the tricks people suggest here. Changing the date on a page without changing the content is noticed and ignored, and it makes your own audit trail useless. If a page deserves attention, give it something new to read; optimizing existing content for AI citation covers what's worth adding.

See which crawlers matter for AI answers

"Crawled" isn't one thing. Different bots do different jobs, and they don't all move you toward being cited.

Three groups, and the difference is practical:

Search crawlers. Googlebot and Bingbot build the indexes that most AI assistants search when they need current information. These visits gate nearly everything downstream, which is why they're the ones to watch when you want an update reflected in answers.

Assistant fetchers. OAI-SearchBot, PerplexityBot, and their equivalents fetch pages while an assistant is putting an answer together. Seeing these on a page is a good sign; it means your page is being retrieved for live questions, not just sitting in an index.

Training crawlers. GPTBot, ClaudeBot, Google-Extended and friends collect content for model training. They affect what a model knows without searching, on a timeline measured in model releases. Useful to see, but not the signal to plan a quarter around.

The pattern to look for is a page that search crawlers visit but assistant fetchers never touch. That usually means the page is indexed and simply isn't being chosen for the questions you care about, which is a content and relevance problem rather than a crawl problem. Adding words to a page that's already read regularly won't fix it.

The reverse pattern is rarer and more urgent. If a page gets almost no visits from any category, check for a block before you check anything else. A stray robots.txt rule or an overzealous bot filter at the CDN can quietly remove a section of your site from everything, and it looks exactly like weak content from the outside. For the full breakdown of who's visiting and what to do about each group, seeing which bots are crawling your site goes deeper than this step needs to.

Time your updates and reports to the real cadence

Now use the number. Two calendars change, and the second one saves you more than the first.

The publishing calendar. Update pages in batches that fit inside one crawl interval, rather than trickling out changes that each wait their turn. If your important pages get refetched every three weeks, shipping eight updates in one week means they all get picked up in roughly the same window, and you get one clean measurement instead of eight overlapping ones. Trickling the same eight changes across two months gives you a blur that nobody can attribute.

The reporting calendar. This is where teams waste the most money and morale. Running a visibility report before the changed pages have been recrawled measures the old version of your site. It costs credits and produces a flat line that gets read as failure.

Match the reporting schedule to the cadence you measured:

Your measured cadence	Run visibility reports	Wait after an update
Daily	Weekly	About 1 week
Weekly	Every 2 weeks	2 to 3 weeks
Every 2 to 4 weeks	Monthly	4 to 6 weeks
Slower than monthly	Monthly, on major changes	6 to 8 weeks

The extra week in each row is the gap between a page being recrawled and the change working through the index and into answers. Being refetched is necessary, not sufficient.

Tell your stakeholders the interval before the work starts, not after a flat report. "We updated the pricing page on the 3rd, our pages get refetched about every three weeks, so we'll have a real read in mid-October" is a sentence that buys patience. The same information delivered after a disappointing report sounds like an excuse.

This also decides what to do when a launch date is fixed and the cadence is slow. You can't speed up the crawl much, so publish the supporting pages earlier instead. Getting a page in place four weeks before the announcement means it's been read by the time anyone asks about it. For how this fits with the rest of the channel's numbers, measuring AI as a marketing channel covers the reporting side.

Verify pickup after the next update

One check closes the loop, and it takes about a minute.

After you publish a change, look at the page's last crawl date. If it's after your publish date, the new version has been read and any measurement from here is fair. If it's before, you're still waiting, and nothing you see in a visibility report tells you anything about the work you did.

Make that check the first step of every post-update review. It reorders the conversation in a helpful way, because "has it been read yet" gets answered before "did it work," and those are different questions with different fixes.

Three outcomes and what each one means:

• Refetched, and the numbers moved. The update worked. Note the interval between publish and pickup; that's your planning number for next time, and it's more reliable than the site-wide average.
• Refetched, and nothing moved. Now you have a content question, and it's a real one. The page was read in its new form and still isn't being chosen. Score it against the query you want and look at what it's missing.
• Not refetched. Not a content question at all. Go back to the starvation causes above, add an internal link from a page that gets crawled often, and check the sitemap entry.

That third outcome is the one worth catching, because it's usually mistaken for the second. Teams rewrite a page two or three times when the problem was that no crawler had been back since the first rewrite. If a page you've updated stays uncrawled for more than double your site average, treat it as a technical issue and run an AI readiness audit on it; blocked crawlers, redirect chains, and content that only appears after JavaScript runs all produce exactly this pattern.

Keep a simple log of publish date, first crawl after publish, and what changed in the numbers. After a quarter you'll have your own lag figure per section of the site, which beats any general advice about how long to wait.

Start with the number you don't have

Pull one figure this week: the average days between crawler visits on your site. You either have server-side crawler logging already, in which case it's a query, or you don't, in which case installing traffic analytics starts collecting it today and gives you a usable number within a few weeks.

That single number changes how you plan. It sets how long to wait before reporting, how to batch updates, how far ahead of a launch to publish, and what to say when someone asks why the dashboard is flat.

The teams that get frustrated with AI visibility work are usually not doing bad work. They're measuring it before anyone has read it.