AI visibility

AI answer engines and search services do not run JavaScript, so browser analytics never sees them. AI visibility records these server-side page fetches for each brand. It stays separate from visitors, sessions, contacts, and revenue, and it never changes how your site responds.

Each fetch lands in one of four categories:

A fetch that went looking for something no crawler reads — a .env file, a key, a debug console, a climb out of your site's folder — or that came from an address its named crawler does not use is counted as suspicious instead. Scanners borrow crawler names all the time, so these are kept out of every category and listed on their own. See Suspicious requests.

Install

npm install @founderhq/events-node

The package needs Node 18 or later.

Next.js middleware

Create the tracker once, then wrap your existing middleware. In Next.js 16 the file is proxy.ts and the export is named proxy; the wrapper is the same.

import { createCrawlerTracker } from "@founderhq/events-node/crawlers";
import {
  NextResponse,
  type NextFetchEvent,
  type NextRequest,
} from "next/server";

const crawlerTracker = createCrawlerTracker({
  secretKey: process.env.FOUNDERHQ_SECRET_KEY!,
  // Who serves your site. See "Behind a CDN or proxy" below.
  proxy: "vercel",
  deliveryMode: "request-scoped",
});

function middleware(_request: NextRequest, _event: NextFetchEvent) {
  return NextResponse.next();
}

export default crawlerTracker.withNextCrawlerTracking(middleware);

If your middleware already redirects, rewrites, or checks authentication, keep that function unchanged, accept its NextFetchEvent, and wrap it in the final export. The adapter uses event.waitUntil() so a low-volume batch is delivered after middleware returns. Pass-through and rewrite status is stored as unknown because middleware cannot observe the final page response. Redirect and thrown error statuses are recorded. The adapter never changes the response.

The /crawlers subpath is safe in Edge middleware and does not import node:crypto.

Middleware runs before Next.js serves a cached or statically generated page, so crawler fetches of those pages are recorded too. A CDN that answers from its own cache is different: the request never reaches your app, and nothing is recorded. See Cached pages.

Behind a CDN or proxy

FounderHQ proves a crawler is real by checking its address against the list its company publishes. The tracker has to know which header holds the visitor's real address — without it, every crawler shows as unverified.

The recorded URL uses the request's Host header. X-Forwarded-Host is only read when a proxy is declared and Host is just an address such as 127.0.0.1:3000, because many proxies pass a visitor's own X-Forwarded-Host through untouched. If your proxy rewrites Host to an internal name, pass resolveUrl and return the public URL.

Express

Add the handler before your routes. It records the final response status when the response finishes.

import express from "express";
import { createCrawlerTracker } from "@founderhq/events-node/crawlers";

const app = express();
const crawlerTracker = createCrawlerTracker({
  secretKey: process.env.FOUNDERHQ_SECRET_KEY!,
});

// Configure Express `trust proxy` for your real proxy topology first.
// The tracker reads Express's resolved req.ip.
app.use(crawlerTracker.expressHandler());

Plain Node

Call trackRequest yourself from any server. It accepts the Node IncomingMessage, a Fetch Request, or any object with method, url, and headers.

import { createServer } from "node:http";
import { createCrawlerTracker } from "@founderhq/events-node/crawlers";

const crawlerTracker = createCrawlerTracker({
  secretKey: process.env.FOUNDERHQ_SECRET_KEY!,
});

createServer((request, response) => {
  response.end("ok");
  crawlerTracker.trackRequest(request, { statusCode: response.statusCode });
}).listen(3000);

process.on("SIGTERM", () => {
  void crawlerTracker.shutdown();
});

trackRequest returns at once. It never throws and never waits on the network.

Any other language

The Node package is a convenience, not a requirement. Any backend can send fetches straight to the ingest endpoint — PHP, Python, Go, Ruby, a Cloudflare Worker, or a job that reads your access logs.

curl https://i.getfounderhq.com/i/v2/crawlers \
  -H "Authorization: Bearer $FOUNDERHQ_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "batch": [
      {
        "url": "https://www.example.com/pricing",
        "userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot",
        "ip": "203.0.113.42",
        "statusCode": 200,
        "occurredAt": "2026-09-16T09:30:00Z"
      }
    ]
  }'

Send up to 50 fetches per request and keep each body under 64 KiB. The endpoint answers 202 with how many fetches it accepted and ignored; anything that is not a crawler is ignored, so you may send every page request. To send less, forward only GET requests for pages whose user agent matches:

bot|crawler|spider|fetcher|indexer|agent|-user|google|mistralai|cohere|grok

That pattern, matched without regard to case, catches every crawler in the catalogue. Batches are limited to 120 a minute per key.

Options

Every tracker also exposes flush(), shutdown(), and getStats() (queued, dropped, delivered).

Delivery guarantees

• Classification runs in memory from the user agent only.
• Batches stay below both 50 records and 60 KiB, then send with keepalive and a 2 second timeout.
• URLs and user agents are bounded before they enter the byte-bounded queue.
• A failed batch is dropped, never retried, and never blocks a request.
• Static assets, /_next/*, /api/*, /i/*, and non-GET requests are skipped before they reach the queue.

What is sent and what is stored

No cookies, headers, bodies, or visitor identity are sent. Individual records are kept for 90 days. Daily summaries power the reports.

Verification

A user agent is only a claim, and scanners borrow crawler names all the time. FounderHQ checks each claim against what the crawler's own company publishes.

Checked against a published address list, refreshed every six hours: OpenAI (ChatGPT-User, OAI-SearchBot, GPTBot), Anthropic (Claude-User, Claude-SearchBot, ClaudeBot), Perplexity (Perplexity-User, PerplexityBot), Google (Googlebot, GoogleOther, Google-InspectionTool, Google-CloudVertexBot, Google-Agent, Gemini Notebook), Microsoft (Bingbot), Apple (Applebot), Mistral (MistralAI-User, MistralAI-Index), DuckDuckGo (DuckDuckBot, DuckAssistBot) and Common Crawl (CCBot).

Checked by reverse DNS, the method the company documents: Amazon's Amazonbot, whose address must resolve to a name under crawl.amazonbot.amazon and back again.

Can't be verified: Meta, ByteDance, xAI, Moonshot (Kimi), Alibaba (Qwen), DeepSeek, Zhipu, Cohere, Allen AI, Baidu, Yandex, You.com, Amazon's Amzn-SearchBot and Amzn-User, and MistralAI-Training. They are still counted under the name they give.

Google-Extended and Applebot-Extended are not crawlers. Google and Apple use them only in robots.txt and never send requests under those names, so a request that names either is not counted as a Google or Apple crawler.

Suspicious requests

A fetch is suspicious when it asks for something no crawler reads, or when its address was checked and is not the named crawler's. It is listed under the name it gave, but counted in none of the categories.

Paths treated as probes include hidden files and folders (except /.well-known), environment files, keys and deploy files, a development server's file route, framework debug consoles, system paths, and attempts to climb out of your site's folder. Routes a real site serves to crawlers — such as /wp-json, /graphql, or any .php page — are not treated as probes.

An address that simply could not be checked is not suspicious on its own: some companies publish no way to check their crawlers, and treating their visits as forgeries would hide the real ones.

Cached pages

A page served from a CDN's own cache never reaches your server, so the tracker never sees that fetch. With a long cache lifetime, most crawler fetches of popular pages can be answered this way, and reports under-count them. Fetches of paths that are never cached, such as probes for missing files, are always recorded — which is one reason suspicious traffic can look larger than it is.

The fix is to let crawler requests through to your server while people keep sharing the cache. Visitors notice nothing, and nothing new is sent anywhere: your server records the fetch the way it does today. The cost is small and comes from crawlers only: each crawler request is one more request to your origin, and on CloudFront each one leaves a cache object nobody reuses.

Amazon CloudFront

A CloudFront Function on the viewer request event runs on every request before CloudFront looks in its cache. The function below marks crawler requests with an x-crawler header whose value is unique per request, and a cache policy puts that header in the cache key, so a crawler's request never matches a cached copy. Visitors carry no header and share the cached copy as before. A visitor cannot use the header to skip the cache: the function deletes whatever the viewer sent before it decides.

1. Create the function. In the CloudFront console open Functions and choose Create function. Name it crawler-cache-key, choose JavaScript runtime 2.0, and choose Continue. On the Build tab replace the sample with this code and choose Save changes.

   // Crawler requests skip the cache; people keep sharing it.
   // Header names in the event are lower-case.
   var CRAWLER_CANDIDATE_PATTERN = /bot|crawler|spider|fetcher|indexer|agent|-user|google|mistralai|cohere|grok/i;
   
   function handler(event) {
     var request = event.request;
     var headers = request.headers;
   
     // Whatever the viewer sent under this name is discarded first, so the
     // header below is always ours.
     delete headers['x-crawler'];
   
     // A request without a User-Agent is nothing we can name, and the server
     // would not record it either. It stays on the shared copy. A request that
     // repeats the header arrives with every copy in multiValue, and the origin
     // reads them joined, so a crawler name in any copy counts here too.
     var agentHeader = headers['user-agent'];
     var userAgent = agentHeader ? agentHeader.value || '' : '';
     if (agentHeader && agentHeader.multiValue) {
       for (var i = 0; i < agentHeader.multiValue.length; i++) {
         userAgent += ' ' + agentHeader.multiValue[i].value;
       }
     }
     if (!userAgent || !CRAWLER_CANDIDATE_PATTERN.test(userAgent)) {
       return request;
     }
   
     // CloudFront gives every production request an id. A console test event
     // may leave it out; a random value keeps the key unique there too.
     var requestId = event.context && event.context.requestId;
     headers['x-crawler'] = { value: requestId || 'test-' + Math.random() };
     return request;
   }

2. Test it. On the Test tab choose the Viewer request event type, add a request header user-agent with the value Mozilla/5.0 (compatible; GPTBot/1.4), and run the test. The output request carries x-crawler. Change the value to a browser's user agent and run it again: no x-crawler.

3. Publish it. On the Publish tab choose Publish function.

4. Associate it. Still on the Publish tab choose Add association. Pick your distribution, the event type Viewer request, and the cache behavior that serves your pages. Choose Add association. Repeat for every behavior that serves pages: the default behavior and any path patterns such as /blog/*. Leave behaviors for hashed assets (/_next/static/*, images, fonts) alone: crawler fetches of assets are not counted, so the header would only cost cache hits there.

5. Create the cache policy. Open Policies, then Cache, and choose Create cache policy. Copy the settings of the policy your page behavior uses today: the three TTLs, query strings, cookies, and compression support. Under Cache key settings → Headers choose Include the following headers and add x-crawler. Choose Create. A managed policy such as CachingOptimized cannot be edited, which is why you copy it.

6. Attach the policy. Open Distributions, choose yours, then the Behaviors tab. Select a page behavior from step 4 and choose Edit. Under Cache key and origin requests keep Cache policy and origin request policy and pick the new policy for Cache policy. Choose Save changes. Repeat for the other page behaviors, then wait until the distribution's status returns to Deployed.

7. Confirm it. Ask for a page twice as a browser, then twice as a crawler.

   # People: the second request is answered from the cache.
   curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36" \
     https://www.example.com/ | grep -i -E '^(x-cache|age):'
   # Crawlers: every request misses, however often you repeat it.
   curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
     https://www.example.com/ | grep -i -E '^(x-cache|age):'

   Browser requests show x-cache: Hit from cloudfront with an age. Crawler requests show x-cache: Miss from cloudfront every time. If they show a hit, the distribution has not finished deploying, or the behavior that served the page still uses the old cache policy.

CloudFront sends every header in the cache key to the origin, so crawler requests arrive with an X-Crawler header. The tracker does not read it. The function is a small script, well under CloudFront's 10 KB limit, with no network access and no imports; it adds no measurable latency, and CloudFront bills it per invocation.

Cloudflare

Cloudflare caches HTML only when a rule tells it to, such as a Cache Rule with Eligible for cache on your pages, or an older Cache Everything Page Rule. Without such a rule crawler fetches reach your server already and nothing is needed. With one, add a Cache Rule that bypasses the cache for crawlers:

1. In the Cloudflare dashboard open your zone, then Rules → Overview, and create a Cache Rule. Name it Crawlers bypass cache.

2. Under When incoming requests match choose Custom filter expression and open the expression editor. On a Business or Enterprise plan, which the matches operator needs, enter:

   (lower(http.user_agent) matches "bot|crawler|spider|fetcher|indexer|agent|-user|google|mistralai|cohere|grok")

   On Free or Pro, matches is not available. Use contains, which is case-sensitive, on the lower-cased agent instead:

   (lower(http.user_agent) contains "bot") or (lower(http.user_agent) contains "crawler") or (lower(http.user_agent) contains "spider") or (lower(http.user_agent) contains "fetcher") or (lower(http.user_agent) contains "indexer") or (lower(http.user_agent) contains "agent") or (lower(http.user_agent) contains "-user") or (lower(http.user_agent) contains "google") or (lower(http.user_agent) contains "mistralai") or (lower(http.user_agent) contains "cohere") or (lower(http.user_agent) contains "grok")

3. Under Cache eligibility choose Bypass cache, then Deploy.

4. Place the rule below the rule that makes pages eligible for cache. When several Cache Rules match one request, the last matching rule wins for a conflicting setting such as bypass versus eligible.

Confirm with the same two curl commands as above, reading cf-cache-status instead: browsers get HIT, crawlers get DYNAMIC, which means Cloudflare did not look in its cache at all.

If your tracker runs in a Cloudflare Worker, add no rule. Workers run before Cloudflare's cache, so the Worker already sees every crawler request; the rule would only send each crawler fetch on to your server as well.

Vercel and Netlify

Nothing is needed. On Vercel, Routing Middleware runs before the cache on every request, so the tracker in your middleware records crawler fetches of cached and statically generated pages. On Netlify, Edge Functions run before the edge cache unless you opt a function into caching its own responses with cache: "manual"; keep the tracker in an ordinary edge function and every request reaches it.

If your CDN is not listed, the same idea applies: a rule that bypasses the cache when the user agent matches the pattern above, or a cache key that varies on a header only crawlers carry.

Choose what appears

Each brand has four switches in Analytics → Settings → AI visibility: AI answer engines, Search indexing, Model training, and Other automated fetches. A switch that is off drops that category at ingest. Existing history stays available. The same card has Create a server key for the tracker.

The SDK can also limit categories at install time:

createCrawlerTracker({
  secretKey: process.env.FOUNDERHQ_SECRET_KEY!,
  enabledCategories: ["AI_ANSWERS", "INDEXING"],
});

What the reports show

The crawler card sits at the bottom of Analytics → Overview and follows the page's period. It has one tab each for AI answers, Indexing, Training, and Suspicious. Each tab counts fetches by service (for example By AI assistant) and by page, so you can see who read what.

A published article matches its published host and path. Verified activity is labelled Verified OpenAI fetches; unverified or unverifiable activity uses qualified wording such as Reported as Anthropic.

Troubleshooting

No hits show up. Confirm the key starts with fhq_sk_ and belongs to the brand you are viewing. Check that the wrapped middleware or handler runs for your public pages (Next.js matcher rules can exclude them). Call getStats() after a known fetch; dropped above zero means delivery failed, queued means the batch has not flushed yet. Remember that the category switches in Settings also drop fetches.

The SDK throws at startup. createCrawlerTracker throws when secretKey is empty or starts with fhq_pk_. Load the secret from the server environment.

403 from the endpoint. The key is publishable, lacks the server events permission, or is disabled. Create a new one with Create a server key in Analytics settings.

Everything shows as unverified. No usable address is reaching the tracker. Pick the setting for your platform in Behind a CDN or proxy; on CloudFront, check that the origin request policy includes CloudFront-Viewer-Address. Do not trust a collection of forwarding headers: a client-reachable origin can otherwise accept a forged crawler address.

Pages show 0.0.0.0 or an internal host. Declare your platform with proxy, which records the public Host. If your proxy rewrites Host to an internal name, pass resolveUrl. Allow outbound HTTPS to i.getfounderhq.com.

AI agent or LLM? Read this page as markdown