SEO tool crawler
Screaming Frog
Crawlers run by SEO platforms, usually reading your site for someone else.
Last updated
At a glance
| Category | SEO tool crawler |
|---|---|
| Operator | Screaming Frog |
| User agent match | contains "screaming frog" |
| Executes JavaScript | No |
| Respects robots.txt | Generally yes |
| Identity verifiable | No, the name is only a claim |
| IVT class | GIVT (general invalid traffic: declared automation) |
What Screaming Frog is
Screaming Frog is operated by an SEO platform. It reads your site to build the reports the platform sells: backlink graphs, keyword data, site audits. Unless you or your agency use that platform, its visits serve a stranger, quite possibly a competitor researching you.
Can the name be trusted?
There is no published way to verify Screaming Frog: the operator provides no IP ranges and no reverse DNS scheme, so the name in the user agent is only a claim. Any script can send it. TrueVisit still names the traffic, and weighs the claim against behavioral signals instead of trusting it.
Should you block it?
Blocking Screaming Frog is a legitimate choice with one caveat: if you or your agency run reports on the platform that operates it, blocking the crawler blocks your own audits. If nobody on your side uses it, there is no downside; it respects robots.txt, and a firewall rule works on the days it forgets.
User-agent: Screaming Frog SEO Spider
Disallow: /For enforcement rather than a polite request, the Cloudflare blocking guide generates copy-paste WAF rules from this same signature list, with guardrails that keep verified search engines safe.
How TrueVisit reports it
Visits from Screaming Frog appear in your bot report under this exact name, never as an anonymous percentage, and every verdict stores the reason it was reached. Because this client never executes JavaScript, a browser-only tracker cannot see it at all; it is visible through server-side collection, which is how the visits reach your report.