AI Access Checker

Is commoncrawl.org blocking AI crawlers?

Spyglasses reads commoncrawl.org's robots.txt and reports, agent by agent, which AI crawlers and assistants it lets in. Check another domain below.

We'll analyze the site's robots.txt for AI crawler restrictions

Want to know how much authority commoncrawl.org carries in the web graph AI models train on?

See commoncrawl.org's Common Crawl rank →

Track how AI describes commoncrawl.org

Spyglasses runs Daily Prompt Tracking across ChatGPT, Gemini, Claude, Perplexity and Google AI Overviews, and monitors every citation those answers hand out, so you can see what commoncrawl.org is credited for and where it is missing.

Checking whether an outlet blocks AI? AIPVS turns that check into a full publisher score.

AI Placement Value Score →

Score the specific article too, landed or still a pitch.

AI Placement Quality Score →

See which publishers AI actually cites in your market.

Citation Intelligence →

PR teams: this check is step one of the whole earned-media toolkit.

For PR & Comms →