AI Agents & Web Crawlers Directory

Bots specifically identified as AI training crawlers. This category specifically refers to AI models designed for broad public use.

Crawlers used to index content for inclusion in private models. Agents in this subcategory usually are managed by a specific company and do not crawl unaffiliated sites.

AI Tools that access data as part of an AI Agent workflow

Direct AI assistants executing web requests

Bots that crawl to improve AI search capabilities

Web archival services

Crawlers from online Ecommerce marketplaces

Major search engine crawlers

Tools for SEO analysis and monitoring

Code modules that can access web pages. Note that it is possible for the application developer to override the user agent, which would cause these tools to not. report correctly.

Visits generated by tools that can be run from a command line, like cURL, wget, and others. These tools are not necessarily malicious, but they are often used for scraping or other data collection, which is why they are listed here.

Known data collection tools

Special Purpose bots that convert web pages into other formats, like screenshots or RSS feeds.

Tools for monitoring website availability and performance

Tools that access a site to render a preview in another application.

Tools for security scanning and monitoring

Tools that monitor the web for instances of brand names and other keywords.

Crawlers from social media platforms

Bots that have not been logged in several years and are likely no longer active.

Bots pending classification