Stop your cache from hiding AI crawlers

If your AI crawler numbers look impossibly low, your cache is probably answering those requests before anything can record them. This is fixable and costs your readers nothing.

Last updated

Why the numbers go missing

AI crawlers do not run JavaScript, so they can only be recorded by your server. If a full page cache answers the request first, your application never runs, and the visit is invisible to everyone including us.

This affects WP Rocket, LiteSpeed, W3 Total Cache, Varnish, and Cloudflare APO equally. It is not a bug in any of them. Serving HTML without running PHP is the entire point of a page cache.

Confirm it is the cache

Compare your server-side pageview count against your browser tracker count. Every browser pageview began as a page request, so the server should see at least as many and normally more. If it sees materially fewer, something answered those requests before your code did. TrueVisit says so on the overview screen rather than quietly reporting the smaller number.

The fix: exclude crawler user agents from the cache

Every serious cache plugin has a user agent exclusion list. Add the crawlers you care about and their requests will reach your application while your readers keep being served from cache.

A reasonable starting list
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Google-Extended
Bytespider

This costs your readers nothing. A crawler fetches a page once and leaves, so it was never going to benefit from your cache, and excluding it does not change what a human is served.

Decide separately whether you want them at all

Seeing crawlers and allowing them are different decisions, and you should make them in that order. Measure first, then choose. Both OpenAI and Anthropic document how to control their crawlers through robots.txt, and Cloudflare offers a one click block if you would rather not manage it yourself.

Blocking blind is the mistake. If you block every AI crawler before you have ever measured them, you will never know what you gave up, and for a lot of publishers being read by an assistant is now a referral channel rather than a theft.

Sources