Server-side collection and AI crawlers
Why a JavaScript tracker cannot see GPTBot or ClaudeBot, how to post events from your own server, and what a full page cache does to the numbers.
Last updated
Why the browser tracker is not enough
AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, and CCBot fetch your HTML and do not execute JavaScript. A browser side analytics script never runs for them, which means they are invisible to Google Analytics, Plausible, Fathom, and to our own tracker.
This is not a limitation we can engineer around inside the browser. The only way to record a request from a client that never runs a script is to record it where the request arrives, on your server.
Posting an event from your server
Create an API key on the install screen, then post one event per HTML request. The full guide with snippets for WordPress, PHP, Node.js, and Python is inside the app, filled in with your own key and URLs.
curl -X POST https://api.truevisit.ai/api/v1/events \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/article",
"referrer": "",
"userAgent": "<the visitor user agent from your server>",
"ip": "<the visitor IP from your server>"
}'Send the visitor IP and user agent, never your server's. Without them we cannot verify that a crawler is who it claims to be, and every hit would appear to come from your own host.
Limits
| Limit | Value |
|---|---|
| Events per batch | 50 |
| Body size per request | 64 KB |
| Events per minute, per site | 10,000, counted per event rather than per call |
| Response to an oversized body | 413, so you can tell size from a malformed payload |
The full page cache problem, stated openly
If your site serves pages from a full page cache such as WP Rocket, LiteSpeed, Varnish, or Cloudflare APO, your application never runs for a cached hit, so the snippet never fires and that request is invisible to us. Most publishers cache, so this is the common case rather than an edge case.
We cannot fix that from our side and we will not pretend the resulting numbers are complete. If your server-side count falls below 80 percent of your browser count, the app says so on the overview screen rather than quietly reporting a smaller number.
The usual fix is to exclude known crawler user agents from your cache, so their requests reach your application while your readers keep being served from cache. It costs your readers nothing, because those clients were never going to benefit from a cache anyway.