Bot Analytics

​

Intro

• AI bots regularly visit websites to collect, update, and verify information not only for the training of LLMs, but also to power real-time AI search and live answers.
• This crawling behavior determines which content is available and ingested into AI chat assistants and how your brand and products are represented in AI-generated responses.
• Importantly, AI bots behave differently than the Google web crawler. Most do not run JavaScript or maintain cookies or session state. If your site is JS heavy or not server-side rendered then there’s a risk that your content is not being ingested correctly.

​

What’s included in the Beta?

1. Identify which types AI bots are accessing your site and how often they visit
2. See which specific pages are being crawled or overlooked
3. Detect patterns, trends, or technical issues that could impact your AI visibility
4. Understand how LLMs and AI chatbot apps are building knowledge about your brand, products and services from your website

​

Implementation

1. Server Log Drain (recommended). Integrated directly with your server logs from Vercel (Netlify, GCP and AWS coming soon).
2. Next.js SDK. Lightweight, easy to install Next.js JavaScript package.
3. HTTP API Endpoint. Send event data manually from any backend via an HTTP POST request.

​

Installation Steps

Vercel

1

Add new 'Custom HTTP endpoint' drain

• In Vercel navigate to Project Settings and select Drains.
• Add a new Custom HTTP endpoint drain.

2

Select "Logs" as data to drain

3

Configure the drain

• Drain Name: GPTrends Bot Analytics
• Projects: Your website project
• Sources: Select all besides builds
• Environment: Production
• Sampling Rate: 100%

4

Set the destination

Add the following POST endpoint URL with your GPTrends API Key appended after ?websiteKey=

Copy

https://api.gptrends.io/v1/intake/vercel-log-drain?websiteKey=[YOUR_GPTRENDS_KEY_HERE]

You can find your GPTrends API Key in the Bot Analytics section of the platform, either in the set up flow or under “Manage Tracking”

5

Select 'Create Drain' to save and finish

Need help? Feel free to grab some time with us here and we’ll be happy to walk you through the set up process

Next.js

1

Step 1

Install the GPTrends tracking package:

Copy

npm install @gptrends/track

2

Step 2

Update your middleware.ts file:

Copy

import { NextResponse } from "next/server";
import type { NextRequest } from "next/server";
import { GPTrendsTrackerNextJS } from "@gptrends/track/nextjs";

const gptrendsTrack = GPTrendsTrackerNextJS({
  websiteKey: "gptrends_secret_...", // Replace with your website key
});

export async function middleware(request: NextRequest) {
  gptrendsTrack(request);
  return NextResponse.next();
}

export const config = {
  matcher: [
    "/robots.txt",
    "/sitemap.xml",
    "/((?!api|_next/static|_next/image|favicon.ico).*)",
  ],
};

Need help? Feel free to grab some time with us here and we’ll be happy to walk you through the set up process

HTTP API

Track bot visits from any backend with a simple GET request:

Copy

GET <https://hffpucuigqemqhzokpvw.supabase.co/functions/v1/track-drain>

Required Parameters:

• url: Full URL accessed
• userAgent: Bot’s User-Agent header
• ref: Referrer header (empty if none)
• ip: Visitor IP
• websiteKey: Your GPTrends key

Example:

Copy

curl "<https://hffpucuigqemqhzokpvw.supabase.co/functions/v1/track-drain?url=https://yourwebsite.com/&userAgent=Mozilla/5.0%20(compatible;%20ChatGPT-User/1.0)&ref=&ip=1.2.3.4&websiteKey=gptrends_secret_your_key_here>"

Notes:

• Only track key pages (blogs, landing pages, robots.txt, sitemap.xml)
• Skip static assets
• Use fire-and-forget requests for zero impact on site speed
• URL-encode all parameters

Need help? Feel free to grab some time with us here and we’ll be happy to walk you through the set up process

​

Security, Privacy & Performance

• No Personal Identifying Information (PII) from users is stored or logged
• The log drain approach occurs outside of the run path and has no impact on your website performance
• The Next.js package runs in a fire-and-forget mode inside middleware; requests are not awaited, so it has no impact on site performance or page load times.
• All errors are safely caught, logged, and discarded — ensuring your site runs exactly as before, with zero interference to end users