Bot directory
Every bot the TrueVisit detection engine names, in one place: what each one is, whether its identity can be verified, and whether blocking it is wise. Generated from the same signature list the engine runs, 170 bots and counting.
Last updated
AI crawlers
Crawlers that read pages to train or ground AI models.
- Ai2Bot
- Amazonbot
- Anthropic AI
- Applebot-Extended
- Bytespider
- CCBot
- Claude-SearchBot
- Claude-Web
- ClaudeBot
- Cohere AI
- Cohere Training Crawler
- Diffbot
- FacebookBot
- Firecrawl
- Google CloudVertexBot
- Google-Extended
- GPTBot
- ImagesiftBot
- img2dataset
- Meta-ExternalAgent
- OAI-SearchBot
- Omgili
- Omgilibot
- PanguBot
- PerplexityBot
- SemanticScholarBot
- Timpibot
- Webzio-Extended
- YouBot
AI assistant fetchers
Fetchers that load a page live because a person asked an AI assistant about it.
Search engine crawlers
Crawlers that index the web for search engines. Blocking them costs rankings.
Ad network crawlers
Crawlers an ad network runs over your pages to decide what to serve on them, or whether to serve at all.
SEO tool crawlers
Crawlers run by SEO platforms, usually reading your site for someone else.
Scrapers
Bulk content copiers. They take your pages and give nothing back.
Security scanners
Probes looking for vulnerable software, or research projects mapping the internet.
Headless browsers
Real browsers driven by scripts. The tooling behind sophisticated automation.
HTTP libraries
Programming language clients. A script somebody wrote, purpose unknown.
Monitoring bots
Uptime and performance checks. Usually configured by you or your own tools.
Link preview bots
Bots that build the preview cards shown when your links are shared.