How to See Which Bots Are Crawling Your Site

To see which bots are crawling your site, you need detection that runs on the server rather than in the browser. Bots request your HTML and leave without executing JavaScript, so a tag-based tool like GA4 never records them. Once you're counting at the server, sort the visits into AI assistants, AI model trainers, search crawlers, specialty bots, and scrapers, then drill into the named agents in each group to see visit counts, unique pages, and last-seen dates. Slice by page to learn what each crawler is after. Then decide who stays welcome.

Bots are usually 30 to 60 percent of all requests to a business site, and almost none of it appears in your analytics. Setting up server-side detection takes under an hour on most stacks, and the first useful picture shows up within a day. AI crawler volume on a typical content site has been climbing faster than any other category, which is exactly why nobody should be guessing at these numbers.

Most teams find out about this the same way. Someone asks how much traffic AI is sending, the marketer opens GA4, filters for referrals from chatgpt.com, and finds a few hundred sessions. That number is real, but it's only the humans who clicked a link in an answer. The much larger story, the machines reading your pages to build those answers in the first place, left no trace at all.

That's the part worth fixing. If you can't see which crawlers reach you, you can't tell whether an AI platform is ignoring your site, reading it constantly and never citing it, or quietly taking your whole archive for training. Those three situations look identical from the outside and need completely different responses.

This guide works with any server-side detection setup; the callouts show how each step runs in Spyglasses.

Step	What you're doing	What tells you it's working
1. Count at the server	Detecting requests before they need a browser	Bot visits appear that GA4 never showed
2. Categorize	Splitting assistants, trainers, search, scrapers	Each category has its own line on the timeline
3. Drill in	Reading the named agents under each category	You can name the top ten bots hitting your site
4. Slice by page	Seeing what each crawler targets	You know which pages draw AI and which get skipped
5. Decide	Blocking, allowing, or reporting	A written rule for each category, not a vibe

Start counting bots at the server

Client-side analytics answers a different question than the one you're asking. GA4 fires when a browser loads and runs its tracking script. A crawler asks for the HTML, parses it, and disconnects. No script runs, no event fires, and the visit is invisible forever.

So detection has to sit in the request path. That means middleware, a CDN worker, a server plugin, or log parsing. Each one sees every request as it arrives, including the ones from software that will never render a page.

The identification itself works off the user agent string plus, for the bots that publish them, verified IP ranges. User agents alone are trivially spoofable, which is why the good detectors cross-check the claim against the address it came from. A request calling itself Googlebot from an IP that Google doesn't own is not Googlebot, and you want to know that.

One thing to settle early: keep bot traffic out of your human reporting. Bots inflate pageviews and wreck bounce rate and time on page, since a crawler that reads and leaves in 80 milliseconds looks like the worst engagement in your account. Count them separately and deliberately.

Split the traffic into categories that mean different things

A raw bot count isn't worth much. Five thousand visits could be great news or a content theft problem, and the difference is entirely in who's behind them. Five categories carry most of the meaning.

AI assistants are the live ones. When someone asks ChatGPT, Claude, Perplexity, or Gemini a question and the assistant decides it needs to read your page to answer, that fetch is an assistant visit. ChatGPT-User is the agent OpenAI sends for this. It's happening on behalf of a person waiting for a response right now. This is the category that connects to revenue.

AI model trainers are bulk collectors. GPTBot, ClaudeBot, Google-Extended, Meta-ExternalAgent, and CCBot gather content for future model training, not for anyone's current question. A trainer visit gives you nothing today. It might make a future model slightly more likely to know who you are, which is a real but slow and unmeasurable benefit.

The GPTBot and ChatGPT-User distinction is the one that trips people up most, because both come from OpenAI and both land in a naive count as the same thing. GPTBot is the training crawler. ChatGPT-User is the live fetch on behalf of someone in a conversation. Blocking OpenAI as a company kills both; blocking GPTBot alone keeps the assistant traffic you want.

Search crawlers are the familiar ones. Googlebot and Bingbot still matter enormously for AI visibility, because ChatGPT and Gemini both run searches against Google's index underneath, and Claude runs against Brave. If Googlebot stops visiting a page, that page drops out of the pool AI assistants pick from.

Specialty bots are SEO tools, social preview fetchers, and monitoring services. Mostly harmless, occasionally noisy, rarely worth attention.

Scrapers are everything else. Some are a developer testing something with curl. Some are systematically copying your catalog.

Category	Example agents	What the visit means	Sensible default
AI assistants	ChatGPT-User, Claude-User, PerplexityBot	Someone asked a question and you were a candidate	Always allow
AI model trainers	GPTBot, ClaudeBot, Google-Extended, CCBot	Bulk collection for a future model	Allow or block, on purpose
Search crawlers	Googlebot, Bingbot	Indexing that feeds AI retrieval too	Always allow
Specialty bots	SEMrush, Ahrefs, social preview fetchers	A tool or platform is checking your page	Allow, ignore
Scrapers	curl, wget, python-requests	Someone is copying content	Block the noisy ones

The split between the first two rows is the one most teams have never seen, and it changes conversations. "AI traffic is up 40 percent" means something very different if it's assistants than if it's one trainer working through your archive.

Drill into the individual bots

Categories tell you the shape. Named bots tell you the story. Open each category and look at the agents underneath, because within one category the individual bots often behave nothing alike.

Four columns do most of the work. Visits is volume. Unique pages is reach, and the ratio between the two is the useful part: 4,000 visits across 40 pages is a bot re-reading a small set constantly, while 4,000 visits across 3,800 pages is one systematic sweep. Last seen catches the crawler that quietly stopped coming three weeks ago, which is a problem you'd otherwise find months late. And the per-visit log settles arguments, because you can look at the actual requests with timestamps and paths.

That ratio is also the fastest way to tell GPTBot from ChatGPT-User without reading a single user agent string. The trainer sweeps wide and shallow. The assistant returns to a narrow set of pages over and over, because those are the pages that keep surfacing in searches.

Watch for a few patterns. A named bot that goes to zero usually means a technical block, a firewall rule, or a robots.txt change nobody announced. A bot you don't recognize showing up with high volume is worth ten minutes of research. And an assistant reading you heavily while your visibility tracking shows no citations is a specific, fixable problem: you're reachable but not quotable, which is what finding and fixing AI citation blockers is about.

See which pages each bot is taking

Now the question the whole workflow exists for. What are they actually reading?

Slice the traffic by page and you usually find something surprising. Trainers tend to sweep broadly and evenly, since their job is coverage. Assistants concentrate hard, because they only fetch what a search surfaced for a real question. That concentration is a gift. The pages assistants keep coming back to are the pages AI considers your answer on those topics, and that list is often not the list your team would have guessed.

Slice by topic or page type next. If your product pages get almost no assistant traffic while your blog gets constant attention, AI has decided you're an information source and not a vendor for those queries. That's a positioning problem showing up in server logs, and it's a lot easier to see here than anywhere else.

Location matters less for strategy but a lot for spotting trouble. Legitimate crawlers run from a small, stable set of data center regions. Sudden volume from somewhere your business has no presence, especially spread thin across thousands of pages, is the signature of a scraper rather than a crawler.

Also look at how often each bot comes back. A page that Googlebot refetches weekly stays current in the index that AI assistants search. A page it hasn't touched in four months is frozen; you can update it today and AI will keep quoting the old version until the recrawl lands.

Decide what to do with what you found

A picture nobody acts on is a screenshot. Every category needs a written decision, and there are only two directions to go.

The first is governance. Blocking model trainers while allowing assistants is the most common setup, and it's the one that needs care, because OpenAI and Anthropic both run separate agents for the two jobs. Block by user agent at the agent level, not by company, or you'll cut off the assistant traffic you want along with the trainer you don't. Disallow GPTBot and keep ChatGPT-User; disallow ClaudeBot and keep the live Claude fetch. Enforcement belongs at the edge, where a blocked request gets a 403 before it touches your content. Robots.txt is a polite request that the bots you most want to stop already ignore.

Whether to block trainers at all comes down to what you sell. If people pay for your content, protect it. If your content exists to sell something else, blocking trainers mostly costs you a little future recognition and buys you very little.

The second direction is reporting. Bot data becomes a channel number when it sits next to the rest of your marketing measurement, and the two halves of AI traffic answer different questions. Crawler visits say AI is reading you. Human clickthroughs from AI answers say it's sending you people. Both belong in the same monthly view, which is the subject of measuring AI as a marketing channel.

One caution on how you read all of this. Crawl volume is a reachability metric, not a visibility metric. An assistant fetching your page means you were a candidate for one answer. Whether you got quoted happens after the fetch, out of sight. To close that loop you need answer-level tracking, which is what an AI visibility report gives you. Crawl data and citation data together tell you which of the two problems you have; either one alone will mislead you.

Start with one day of data

You don't need a project plan for this. Install detection, wait 24 hours, and look at what shows up. Almost everyone finds at least one thing they didn't expect, usually a trainer working through the archive at a volume nobody had any idea about, or a search crawler that stopped visiting a section of the site months ago.

That first look is worth more than the dashboard you'll build later, because it turns an argument about AI into a list of named bots and page counts. From there the workflow is a ten minute check once a week, and the traffic that used to be invisible turns into the earliest signal you have about whether AI can find you at all.