Server-side collection and AI crawlers

Why the browser tracker is not enough

AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, and CCBot fetch your HTML and do not execute JavaScript. A browser side analytics script never runs for them, which means they are invisible to Google Analytics, Plausible, Fathom, and to our own tracker.

This is not a limitation we can engineer around inside the browser. The only way to record a request from a client that never runs a script is to record it where the request arrives, on your server.

Posting an event from your server

Create an API key on the install screen, then post one event per HTML request. The full guide is inside the app, filled in with your own key and URLs, with ready snippets for WordPress, Cloudflare Workers, Next.js, Remix, Nuxt, SvelteKit, Astro, Laravel, plain PHP, Express, Fastify, Go, Rails, Django, and FastAPI. Shopify and Ghost get the browser tracker only, because neither lets you run code per request, which means neither can report an AI crawler; putting a CDN you control in front and using the Cloudflare Workers snippet is the way around that. If your stack is not on that list, the Events API is one authenticated POST, so any language that can make an HTTPS request already works, and we will write the snippet for you if you ask.

curl -X POST https://api.truevisit.ai/api/v1/events \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "referrer": "",
    "userAgent": "<the visitor user agent from your server>",
    "ip": "<the visitor IP from your server>"
  }'

Next.js App Router

The App Router has no app.use and no next(), so an Express middleware example cannot be pasted into it with the names changed. Save this as middleware.ts in your project root, beside app/. It works on both the Edge and Node runtimes and needs nothing installed.

import { NextResponse } from 'next/server';
import type { NextFetchEvent, NextRequest } from 'next/server';

const TRUEVISIT_URL = 'https://api.truevisit.ai/api/v1/events';
const TRUEVISIT_KEY = process.env.TRUEVISIT_API_KEY;

// A crawler sends no Sec-Fetch-Dest at all, so an absent header counts. A real
// navigation is 'document'. An App Router route change is an RSC request that
// is not a prefetch. Everything else is an XHR, an image, or a stylesheet.
function isPageRequest(request: NextRequest): boolean {
  const dest = request.headers.get('sec-fetch-dest');
  if (dest === null || dest === 'document') return true;
  const isPrefetch = request.headers.get('next-router-prefetch') === '1';
  return request.headers.get('rsc') === '1' && !isPrefetch;
}

export function middleware(request: NextRequest, event: NextFetchEvent) {
  if (!TRUEVISIT_KEY || !isPageRequest(request)) return NextResponse.next();

  const body = {
    url: request.url,
    referrer: request.headers.get('referer') ?? '',
    userAgent: request.headers.get('user-agent') ?? '',
    ip:
      request.headers.get('cf-connecting-ip') ??
      request.headers.get('x-real-ip') ??
      request.headers.get('x-forwarded-for')?.split(',')[0]?.trim() ??
      '',
  };

  event.waitUntil(
    fetch(TRUEVISIT_URL, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Bearer ${TRUEVISIT_KEY}`,
      },
      body: JSON.stringify(body),
      signal: AbortSignal.timeout(2000),
    }).catch(() => {}),
  );

  return NextResponse.next();
}

export const config = {
  matcher: ['/((?!_next/static|_next/image).*)'],
};

Do not add a check that skips bots. Filtering invalid traffic is our job, and crawlers are exactly the traffic this path exists to capture. Note also that request.ip was removed in Next 15, which is why the IP is read from headers instead.

Limits

The full page cache problem, stated openly

If your site serves pages from a full page cache such as WP Rocket, LiteSpeed, Varnish, or Cloudflare APO, your application never runs for a cached hit, so the snippet never fires and that request is invisible to us. Most publishers cache, so this is the common case rather than an edge case.

We cannot fix that from inside your application, and we will not pretend the resulting numbers are complete. If your server-side count falls below 80 percent of your browser count, the app says so on the overview screen rather than quietly reporting a smaller number.

There is one integration that does not have this problem, and it is the reason Cloudflare Workers is on the platform list. A Worker runs at the edge on every request matching its route, including the ones Cloudflare answers from its own cache without ever waking your origin. If you are already proxying through Cloudflare and you cache aggressively, that is the snippet to use: it sees the traffic your application never will.

The usual fix is to exclude known crawler user agents from your cache, so their requests reach your application while your readers keep being served from cache. It costs your readers nothing, because those clients were never going to benefit from a cache anyway.