How to Govern AI Crawlers on Your Site Without Losing AI Visibility

Governing AI crawlers means deciding, on purpose, which bots get to read your site, then enforcing that decision somewhere it holds. Doing it without hurting your AI search visibility comes down to one rule; block agents by name, never whole companies. It takes seven steps and about two hours of real work spread over a couple of weeks.

Start by counting what already reaches you at the server, since bots don't run JavaScript and never appear in your analytics tool. Sort those visits into five categories and write one rule per category. Then get specific, because blocking a company by name is the mistake that costs teams real traffic; OpenAI runs GPTBot for training and ChatGPT-User for live answers, and a company-wide block kills both. State your policy in robots.txt for the crawlers that honor it, and enforce it at the edge for the ones that don't. Watch the blocked requests for a week to confirm each rule matched what you meant. Then check that search crawlers, assistant visits, and your citation counts held steady. Recheck the whole thing quarterly.

Here's the version most teams live through. Someone forwards an article about AI companies scraping content, a rule goes into robots.txt that afternoon blocking everything with "GPT" in the name, and nobody looks at it again. Four months later the traffic from ChatGPT answers is down and nobody can say why, because the block that was supposed to stop training also stopped the assistant that was sending customers.

Why a crawler policy is a marketing decision

It gets treated as an IT chore, and that's how it goes wrong.

The people who set crawler rules are usually the ones with access to the CDN. The people who feel the results are the ones responsible for pipeline. Those are rarely the same team, and the gap between them is where a two-line robots.txt change quietly turns off a channel.

The stakes have changed too. Blocking a crawler used to mean choosing whether a search engine indexed you. Now the same decision covers four or five different things at once: whether a model gets trained on your writing, whether an assistant can read your page while a buyer waits for an answer, whether your pages stay in the search index that assistants query underneath, and whether somebody can lift your catalog wholesale. One toggle labeled "block AI bots" collapses all of that into a single yes or no, and there's no version of that answer that's right for all four.

The fix isn't complicated. It's a written policy with a rule per category, enforced where a request can actually be stopped, and checked after the fact. This guide works with any server-side detection and edge setup; the callouts show how each step runs in Spyglasses.

Step	What you're doing	What tells you it's working
1. Take inventory	Counting the bots you actually get	A named list, not a guess
2. Write the rules	One decision per category	Every category has a reason attached
3. Get agent specific	Splitting trainers from assistants	Rules name agents, not companies
4. Enforce	Robots.txt plus edge blocking	Blocked agents get a 403
5. Watch the 403s	Confirming the rules matched	Blocked visits fall, allowed ones don't
6. Check what you kept	Verifying nothing good got caught	Search and assistant traffic flat
7. Review	Rechecking quarterly	New agents get a rule within a month

Take inventory of the crawlers already reaching you

You can't govern what you can't count, and right now you probably can't count any of it. Bots request your HTML and leave without running JavaScript, so a tag-based tool like GA4 never records the visit. The traffic exists only in your server logs.

So put detection in the request path first. Middleware, a CDN worker, a server plugin, or log parsing all work. Then wait. Two weeks of data is the right amount before you write a single rule, because one week can miss a monthly sweep and anything shorter turns into a policy about whichever bot happened to show up on Tuesday.

While that runs, look at two numbers per bot. Visits tells you volume. Unique pages tells you reach. The ratio between them is the tell you'll use constantly: 5,000 visits across 4,600 pages is a systematic sweep, while 5,000 visits across 30 pages is something re-reading a small set because those pages keep surfacing in searches. The first is usually a trainer. The second is usually an assistant, and it's a list worth keeping.

If none of this exists yet at your company, set it up first and come back; seeing which bots are crawling your site covers the detection side end to end.

Write one rule for each category of bot

Now make the decisions, before you open any settings screen. Five categories cover almost everything, and only one of them is a real judgment call.

AI assistants fetch a page because someone is waiting on an answer. Allow them. This is the category that turns into customers, and blocking it is the most expensive mistake available here.

Search crawlers feed the index that AI retrieval runs against. Googlebot and Bingbot aren't optional. ChatGPT and Gemini both search against Google underneath, so a page that drops out of the search index drops out of the pool assistants pick from.

AI model trainers are the judgment call. They collect in bulk for a future model, which does nothing for you today.

Specialty bots are SEO tools, uptime monitors, and social preview fetchers. Allow them and stop thinking about it.

Scrapers are everything unnamed. Block the noisy ones.

Category	Example agents	What you get from the visit	Sensible default
AI assistants	ChatGPT-User, Claude-User, PerplexityBot	A live answer you might be quoted in	Allow
Search crawlers	Googlebot, Bingbot	Index presence that feeds AI retrieval	Allow
AI model trainers	GPTBot, ClaudeBot, Google-Extended, CCBot	A future model may recognize you	Decide deliberately
Specialty bots	SEO tools, monitors, preview fetchers	A tool is checking your page	Allow
Scrapers	curl, wget, unnamed agents	Usually someone copying content	Block the noisy ones

For the trainer row, the question is simple. Is your content the product, or is it marketing for the product? If people pay for your writing, protect it. If your content exists to sell software or services, a trainer block buys you very little and costs you a little future recognition.

Write the reason next to each decision, put a name on it, and date it. Six months from now somebody will ask why GPTBot is blocked, and "it was in the config when I got here" is how a policy becomes permanent by accident.

Block by agent, not by company

This is the step that separates a policy that works from one that costs you traffic, and it comes down to one fact. The big AI companies run several crawlers under separate names, and they do completely different jobs.

Company	Training crawler	Live assistant fetch	What a company-wide block costs
OpenAI	GPTBot	ChatGPT-User	Every ChatGPT answer that would have read your page
Anthropic	ClaudeBot	Claude-User	Live Claude fetches on your pages
Google	Google-Extended	Googlebot	Search rankings and AI Overviews together
Perplexity	PerplexityBot	Perplexity-User	Answers in a search-first assistant

Google is the one that punishes a careless rule hardest. Google-Extended is a training control and has no effect on search. Googlebot is the index. Block Googlebot and you've turned off search rankings and the retrieval layer that other assistants query, all from a line meant to keep your content out of a training set.

So write rules against user agent names, one at a time. "Block OpenAI" is not a rule you can safely implement. "Disallow GPTBot, allow ChatGPT-User" is.

Then confirm your setup can express the exception, because rule priority is what makes this work. A specific agent rule has to override a category block, or blocking the trainer category takes the assistant with it the moment a company reshuffles which agent sits where.

Put the rules where they can actually be enforced

There are two places a crawler rule can live, and you want both, for different reasons.

Robots.txt is a public statement of intent. The major named crawlers generally honor it, and it's the file people check when they want to know your stance. It costs nothing and it handles the well-behaved majority.

Edge enforcement is the part that holds. A rule at your CDN, middleware, or worker layer returns a 403 before the request ever touches your content. This is what stops the crawlers that read robots.txt and ignore it, which is most of the ones you'd actually want stopped. A scraper systematically copying your catalog was never going to respect a text file.

Put the same policy in both places. If they disagree, you'll spend an afternoon next quarter figuring out which one is real.

Two details are worth getting right on the way in. Exclude the paths that shouldn't be governed at all, like health check endpoints, admin routes, and internal APIs, so a broad rule can't take down monitoring. And make sure the rules can change without a deploy. A policy that requires an engineering ticket to adjust is a policy that stops getting adjusted, usually right after the first time it blocks something important.

Watch the blocked requests for a week

A rule you haven't verified is a guess with a config file attached. Give it a week and look at four things.

Did the blocked agents stop? A working block shows that agent's allowed visits going to near zero within a day or two, with a matching rise in blocked requests. If the visits keep landing, your rule isn't matching the string that agent actually sends.

Did anything unexpected start collecting 403s? This is the one worth setting an alert on. A browser user agent showing up in the blocked list means a real person hit a wall, and that's a small outage you'd rather find in week one.

Did the allowed categories hold flat? Assistant and search crawler volume should look exactly like it did before the change. Any dip there means a category rule caught something a specific rule was supposed to protect.

Who's still knocking? Some agents keep hammering a 403 for weeks. That's fine, and it's also a useful signal, since a crawler that keeps trying after a clear disallow was never honoring robots.txt in the first place. Now you know which of your rules were load-bearing.

Check that nothing you wanted kept got caught

Blocked traffic going down is the easy half. The half that actually matters is proving you didn't break anything, and it takes a little longer to show up.

Start with search. Confirm Googlebot and Bingbot fetch counts held steady, and check Search Console for new crawl errors. A drop here is the most expensive possible outcome of this workflow, and it's silent for weeks if nobody looks.

Then check recrawl cadence on your important pages. Crawlers that get blocked or slowed stop refreshing your content, and a page that isn't refetched keeps getting quoted at whatever it said last time you were crawled. That lag is worth understanding either way; timing content updates to AI recrawl cadence goes into it properly.

Then check visibility itself, which is the part crawler data can't tell you. An assistant fetching your page means you were a candidate for one answer. Whether you got quoted happens after the fetch, out of sight of any traffic tool. Run your visibility tracking before the rule change and again a month after, and compare citation counts and share of voice. If citations fall while assistant visits stay flat, the cause is somewhere else, and an AI visibility report is where you'd see it.

Give this a month. Crawl effects show up in days, retrieval effects show up in weeks, and calling it clean after 48 hours is how a slow problem gets missed.

Review the policy on a schedule

Crawler policy is the kind of thing that gets set once during a scare and then runs unattended for years. Put a recurring 20 minutes on the calendar every quarter instead.

Four things to check each time.

1. New agents. New crawlers show up constantly, and an unrecognized agent with real volume is either a company you've never heard of or a scraper. Both deserve a rule.
2. Renamed agents. Companies rename and split their crawlers without much notice, and a rule pointed at a name that no longer exists reads as active while enforcing nothing.
3. Your own stance. Launching a paywall, a research report, or a customer community changes the trainer answer. So does a strategy shift toward AI as a lead source.
4. What it cost you. Compare assistant visits and citations against the quarter before the rules went in. If a block bought nothing measurable, it's fair to reverse it.

Keep a short changelog with the date, the rule, and the reason. Annotate the change on your visibility timeline too, so when someone asks in March why AI traffic shifted in November, the answer is attached to the chart instead of living in somebody's memory.

Start with the two rules nobody argues about

If this feels like a lot for something that used to be a one-line file, start smaller. Allow the assistants. Allow the search crawlers. Those two rules are uncontroversial, they protect the traffic that turns into customers, and they're the ones most likely to be broken right now by a well-intentioned block somebody added last year.

Everything else can wait for the two weeks of inventory data. The trainer decision is the only real judgment call in here, and it deserves an actual conversation rather than a toggle flipped on a Friday.

The habit is ten minutes a month looking at the blocked list and the allowed categories side by side. The payoff is that when someone senior asks whether AI companies are taking your content, the answer is a policy with names and dates on it, and you can say exactly what you allowed and what it earned you.