Will blocking these bots hurt my Google rankings?

No. Verified search engine crawlers are never offered as an option in the rule builder, and the impersonator rule explicitly excludes anything Cloudflare verifies as a known good bot, so real Googlebot and Bingbot cannot be caught by it.

Is robots.txt enough to stop these crawlers?

robots.txt is a request and a Cloudflare rule is a wall. Reputable SEO tools honor robots.txt; scrapers and impersonators ignore it. Use robots.txt for politeness and the WAF rule for enforcement.

Should I use Block or Managed Challenge?

Block for crawlers that plainly identify themselves and have no business on your site. Managed Challenge for operators that change behavior over time: a challenge fails bots while never hard-locking a person who was misidentified.

Will this fix my inflated analytics numbers?

It removes the bots that announce themselves in their user agent. Bot traffic that fakes a real browser and rotates addresses cannot be caught by any user agent rule; that is what traffic verification and source-level blacklisting exist for.

Does the Cloudflare Free plan support this?

Yes. The Free plan includes five active custom rules and this guide uses at most three: one Block rule for SEO tools and scrapers, one Block rule for search engine impersonators, and one Managed Challenge rule for AI data collectors.

Should I block GPTBot, ClaudeBot, and PerplexityBot?

That is a real trade, not hygiene. These crawlers are how ChatGPT, Claude, and Perplexity read and then cite your site, so blocking them removes you from AI answers. The builder leaves them unticked by default and lets you decide.

How do I undo a rule that blocked something I needed?

Open Security rules in Cloudflare, find the rule by the name you gave it, and toggle it off or delete it. The effect ends immediately; nothing is cached.

How do I know the rules worked?

Open your realtime view before deploying and note which named bots appear. Within minutes of deploying, the blocked names stop appearing while your human pageviews, reading time, and scroll depth stay exactly where they were.

Block spam bots, scrapers, and AI crawlers in Cloudflare

Your report says most of your traffic is crawlers. Here is the five minute version of acting on it, built for affiliate marketers and SEO publishers: tick the bots you want gone, copy the generated expression into Cloudflare, and watch them vanish from your realtime view. The guardrails keep you from blocking anything you would regret.

Last updated Aug 29, 2026

Why block them at all

They cost you real money: every crawler hit is bandwidth and server time you pay for, and on a busy content site the junk share runs past half of all requests.
They distort the numbers an affiliate marketer buys with. A zone that looks alive because scrapers hammer it gets budget a dead zone deserves.
For a publisher, invalid traffic is an AdSense account risk, not just a statistics problem.
AI training collectors take your content into models and send nothing back, unlike the AI assistants that cite you.
Junk buries signal: the fewer machines in your logs and reports, the faster you spot the traffic that actually matters.

What this will and will not touch

These rules block crawlers that announce themselves in their user agent: SEO tools reading your site for somebody else, scrapers, and AI training collectors. Three things are protected by how the rules are built, not just by advice. Verified search engines are never offered as an option, and the impersonator rule below explicitly excludes anything Cloudflare has verified as a known good bot. AI assistants like GPTBot and PerplexityBot start unticked, because they are how your site ends up cited in AI answers, and blocking them is a real trade, not hygiene. Monitoring and link preview bots are not offered at all, since blocking those breaks your own uptime checks and social shares.

One thing no user agent rule can catch: bot traffic that fakes a real browser and never runs JavaScript. Those show up in your TrueVisit reports as suspected bots with the "our JavaScript never ran" reason, and the tool for them is the attribution blacklist, not a Cloudflare rule.

Build your rules

Everything below is generated from the same signature list the TrueVisit detection engine runs, so the names here are the names in your bot report. Tick what you want gone.

Fed by the same signature list the TrueVisit detection engine runs: 172 signatures, refreshed weekly. Expressions verified against Cloudflare Free.

Also block fake Googlebot and Bingbot: user agents claiming to be a search engine that Cloudflare has not verified. The real ones are excluded by `cf.client.bot`, so this rule cannot touch them.

Rule 1 · 28 crawlers · action: Block

(lower(http.user_agent) contains "ahrefssiteaudit") or
(lower(http.user_agent) contains "splitsignalbot") or
(lower(http.user_agent) contains "screaming frog") or
(lower(http.user_agent) contains "dataforseobot") or
(lower(http.user_agent) contains "siteauditbot") or
(lower(http.user_agent) contains "contentking") or
(lower(http.user_agent) contains "serpstatbot") or
(lower(http.user_agent) contains "zoominfobot") or
(lower(http.user_agent) contains "similarweb") or
(lower(http.user_agent) contains "linkdexbot") or
(lower(http.user_agent) contains "semrushbot") or
(lower(http.user_agent) contains "barkrowler") or
(lower(http.user_agent) contains "linkpadbot") or
(lower(http.user_agent) contains "ahrefsbot") or
(lower(http.user_agent) contains "seobility") or
(lower(http.user_agent) contains "megaindex") or
(lower(http.user_agent) contains "webcopier") or
(lower(http.user_agent) contains "rogerbot") or
(lower(http.user_agent) contains "sitebulb") or
(lower(http.user_agent) contains "seokicks") or
(lower(http.user_agent) contains "oncrawl") or
(lower(http.user_agent) contains "mj12bot") or
(lower(http.user_agent) contains "blexbot") or
(lower(http.user_agent) contains "sistrix") or
(lower(http.user_agent) contains "httrack") or
(lower(http.user_agent) contains "dotbot") or
(lower(http.user_agent) contains "scrapy") or
(lower(http.user_agent) contains "wpbot")

Rule 2 · search engine impersonators · action: Block

((lower(http.user_agent) contains "googlebot") or
(lower(http.user_agent) contains "bingbot")) and not cf.client.bot

Rule 3 · 22 AI crawlers · action: Managed Challenge

(lower(http.user_agent) contains "cohere-training-data-crawler") or
(lower(http.user_agent) contains "google-cloudvertexbot") or
(lower(http.user_agent) contains "meta-externalfetcher") or
(lower(http.user_agent) contains "meta-externalagent") or
(lower(http.user_agent) contains "semanticscholarbot") or
(lower(http.user_agent) contains "webzio-extended") or
(lower(http.user_agent) contains "duckassistbot") or
(lower(http.user_agent) contains "imagesiftbot") or
(lower(http.user_agent) contains "facebookbot") or
(lower(http.user_agent) contains "img2dataset") or
(lower(http.user_agent) contains "bytespider") or
(lower(http.user_agent) contains "amazonbot") or
(lower(http.user_agent) contains "cohere-ai") or
(lower(http.user_agent) contains "omgilibot") or
(lower(http.user_agent) contains "firecrawl") or
(lower(http.user_agent) contains "timpibot") or
(lower(http.user_agent) contains "pangubot") or
(lower(http.user_agent) contains "diffbot") or
(lower(http.user_agent) contains "omgili") or
(lower(http.user_agent) contains "youbot") or
(lower(http.user_agent) contains "ai2bot") or
(lower(http.user_agent) contains "ccbot")

Paste them into Cloudflare

Open your domain in the Cloudflare dashboard and go to Security, then Security rules, then Create rule (older dashboards call the same screen WAF, then Custom rules).
Give the rule a name you will recognize next month, for example "TrueVisit: SEO and scraper crawlers".
Click Edit expression and paste the generated text for Rule 1.
Set the action to Block for Rules 1 and 2, and Managed Challenge for Rule 3, then Deploy.
Repeat for each rule you generated. The Free plan allows five active custom rules; this page spends at most three.

Rule 3 uses Managed Challenge instead of Block on purpose: crawler operators change behavior over time, and a challenge fails bots while never hard-locking a person who was misidentified.

Watch it work, live

Before you deploy, open your site in TrueVisit and go to the Realtime tab. Note which named bots are in "Who is reading it".
Deploy the rules.
Within minutes, the blocked names stop appearing in the realtime window, while the human line stays exactly where it was.
A day later, open the Bots tab and compare: the crawlers you blocked flatline from the deploy hour.

That last check matters more than it looks. The proof that you blocked only machines is that your human pageviews, reading time, and scroll depth did not move.

When not to block

You run Ahrefs or Semrush audits on your own site: blocking their bots blocks your own reports. Untick the tool you use.
You are an agency and a client audits this site with one of these tools. Ask before you wall it off.
A staging site shares the Cloudflare zone: rules apply zone-wide, so a block here is a block there too.

If you prefer asking to blocking

Most SEO crawlers and the reputable AI collectors respect robots.txt. Asking is reversible and polite, and it also does nothing against anyone who ignores it, which is exactly the difference between robots.txt and a Cloudflare rule.

robots.txt lines for the polite subset

User-agent: SemrushBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

Frequently asked questions

Will blocking these bots hurt my Google rankings?: No. Verified search engine crawlers are never offered as an option in the rule builder, and the impersonator rule explicitly excludes anything Cloudflare verifies as a known good bot, so real Googlebot and Bingbot cannot be caught by it.
Is robots.txt enough to stop these crawlers?: robots.txt is a request and a Cloudflare rule is a wall. Reputable SEO tools honor robots.txt; scrapers and impersonators ignore it. Use robots.txt for politeness and the WAF rule for enforcement.
Should I use Block or Managed Challenge?: Block for crawlers that plainly identify themselves and have no business on your site. Managed Challenge for operators that change behavior over time: a challenge fails bots while never hard-locking a person who was misidentified.
Will this fix my inflated analytics numbers?: It removes the bots that announce themselves in their user agent. Bot traffic that fakes a real browser and rotates addresses cannot be caught by any user agent rule; that is what traffic verification and source-level blacklisting exist for.
Does the Cloudflare Free plan support this?: Yes. The Free plan includes five active custom rules and this guide uses at most three: one Block rule for SEO tools and scrapers, one Block rule for search engine impersonators, and one Managed Challenge rule for AI data collectors.
Should I block GPTBot, ClaudeBot, and PerplexityBot?: That is a real trade, not hygiene. These crawlers are how ChatGPT, Claude, and Perplexity read and then cite your site, so blocking them removes you from AI answers. The builder leaves them unticked by default and lets you decide.
How do I undo a rule that blocked something I needed?: Open Security rules in Cloudflare, find the rule by the name you gave it, and toggle it off or delete it. The effect ends immediately; nothing is cached.
How do I know the rules worked?: Open your realtime view before deploying and note which named bots appear. Within minutes of deploying, the blocked names stop appearing while your human pageviews, reading time, and scroll depth stay exactly where they were.