See Which AI Bots Crawl Your Site: Self-Hosted Analytics for GPTBot, ClaudeBot and PerplexityBot

To see which AI bots crawl your site, read your web server's access log: every request from GPTBot, ClaudeBot, PerplexityBot and the rest is written there with its IP address, page and status code, even though these bots never run the JavaScript that Google Analytics or Plausible depend on. A few lines of awk count requests per bot, GoAccess turns the same log into an HTML report with AI bots in their own categories, and a 60-line Python script checks each IP against the lists that OpenAI, Anthropic, Perplexity, Google, Bing, Apple and Common Crawl publish, so a scanner that only pretends to be GPTBot gets flagged. This is self-hosted AI crawler analytics: the same questions Cloudflare's AI Crawl Control answers, on your own VPS, with no third party in the middle.

Key facts, checked on 13 October 2026:

• 52% of crawler requests on Cloudflare's network were for AI training in June 2026, up from 22% in spring 2025 (Cloudflare, 1 July 2026).
• Cloudflare reported on 30 September 2026 that more than half of the internet traffic it sees is no longer human, and that daily requests from AI agents grew by more than 1,700% in a year (Cloudflare).
• OpenAI publishes four IP lists, one per bot (the fourth, OAI-AdsBot, only checks pages submitted as ChatGPT ads). The ChatGPT-User list held 234 IPv4 prefixes and the GPTBot list 18 when we downloaded them today (OpenAI bots page).
• Anthropic publishes one list for all its bots at claude.com/crawling/bots.json: 38 IPv4 prefixes, dated 7 October 2026 (Anthropic help center).
• Google-Extended will never appear in your log. Google says it "doesn't have a separate HTTP request user agent string" (Google Search Central).
• In our lab, awk summarised a 1,000,100-line, 178 MB access log in 1.45 seconds, GoAccess in 14.2 seconds, and none of the tools used more than 16 MB of RAM.

Three kinds of AI bot, and why the difference matters

Before counting anything, sort the bots into three groups. They behave differently, they follow robots.txt differently, and they mean different things for your traffic.

• Training crawlers fetch pages in bulk to build datasets for future models. GPTBot, ClaudeBot, CCBot (Common Crawl) and meta-externalagent are in this group. They send you nothing back directly.
• Search crawlers build the index an AI assistant searches when it answers a question. OAI-SearchBot, Claude-SearchBot and PerplexityBot belong here. Block them and you disappear from those answers, the same way blocking Googlebot removes you from Google.
• User fetchers visit one page because a person asked the assistant about it right now: ChatGPT-User, Claude-User, Perplexity-User and meta-externalfetcher. OpenAI, Perplexity and Meta all say robots.txt may not apply to these, because a human started the request.

A user fetch is the closest thing to a reader you will get from an AI assistant. A training crawl is closer to a photocopier. Mixing them in one number, as many dashboards do, hides the part you care about.

Who is who: user agents and how to check them

A user agent is the name a client sends with each request. Anyone can type any name, so each operator also tells you how to confirm that a request really came from them: a published IP list, a reverse DNS name, or both. This table is from the official pages, checked on 13 October 2026, and the IP counts are from the files we downloaded the same day.

Sources: OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl and Meta. The Bing count comes from the JSON file itself.

Two names in robots.txt guides never show up in logs. Google-Extended and Applebot-Extended are robots.txt tokens only: they tell Google and Apple whether pages their normal crawlers already fetched may be used for AI training. Apple puts it plainly: "Applebot-Extended does not crawl webpages". If you want to set those rules, our guide to Content Signals, robots.txt and llms.txt covers them.

Look at the dates inside the files, too. Each JSON file has a creationTime field. Bing's list we downloaded today was created on 3 January 2024 and PerplexityBot's on 7 February 2025, while Google's was regenerated the day before. An old date does not make a list wrong, but it explains why reverse DNS is still the better test for Google, Bing, Apple and Common Crawl.

Our lab setup

We ran everything below in a Debian 13 container limited to 1 GiB of RAM and 2 vCPUs, on one of our Amsterdam servers. It ran nginx 1.26.3, GoAccess 1.9.3, jq 1.7 and Python 3.13.5 from the Debian repositories.

# as root
apt install -y nginx goaccess curl jq python3 bind9-dnsutils
useradd -m -s /bin/bash -G adm loguser

On Debian, nginx writes /var/log/nginx/access.log owned by www-data with group adm and mode 0640. Putting your analysis user in the adm group lets it read the log without root. Everything after this runs as that normal user.

We had no real AI traffic to point at a lab, so we replayed it. A 35-line script sent 685 requests with the exact user agent strings from the vendor pages, from IP addresses inside each vendor's published ranges, plus a few fakes and some human visits with a chatgpt.com or perplexity.ai Referer. To make nginx log those IP addresses, the lab site trusted an X-Forwarded-For header from localhost, which is the same setting you use behind a reverse proxy or CDN. One of the requests looked like this:

# as loguser
curl -s -o /dev/null -H "X-Forwarded-For: 198.51.100.7" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  http://127.0.0.1/.env

On a real server you skip all of this and point the commands at the log you already have. If your site sits behind Cloudflare or another proxy, make sure nginx logs the visitor's IP and not the proxy's, or every verification below will fail. Our reverse proxy guide shows the setup.

Report 1: awk one-liners

The default nginx and Apache "combined" log format puts the request, the Referer and the user agent in double quotes. Splitting each line on the quote character with -F'"' gives you the request in field 2, the status and size in field 3, the Referer in field 4 and the user agent in field 6.

Requests per bot

# as loguser
awk -F'"' 'match($6, /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|meta-externalagent|Googlebot|bingbot|Applebot/) { print substr($6, RSTART, RLENGTH) }' \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn

    183 ClaudeBot
    151 GPTBot
     86 Googlebot
     37 OAI-SearchBot
     34 PerplexityBot
     25 bingbot
     24 ChatGPT-User
     20 meta-externalagent
     15 CCBot
     14 Claude-SearchBot
     12 Applebot
      9 Claude-User
      7 Perplexity-User

Our first try used grep -oi and doubled several counts. The reason: GPTBot's user agent also contains the link https://openai.com/gptbot, so a case-insensitive grep matches it twice per line. The awk version prints only the first match per line, which gives one count per request.

Top pages AI bots read, with status codes

# as loguser
AI='gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|claude-searchbot|perplexitybot|perplexity-user|ccbot|meta-externalagent'
awk -F'"' -v ai="$AI" 'tolower($6) ~ ai { split($2, r, " "); split($3, s, " "); print s[1], r[2] }' \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -12

    154 200 /blog/vps-guide/
    130 200 /blog/
     40 404 /tag/linux/page/9/
     40 200 /
     34 200 /vps-server/
     23 404 /.env
     22 200 /docs/setup/
     20 404 /old-pricing/
     15 200 /robots.txt
      9 404 /wp-login.php
      7 404 /.git/config

404s per bot

# as loguser
awk -F'"' -v ai="$AI" 'tolower($6) ~ ai { split($3, s, " "); if (s[1] == 404) { split($2, r, " "); match(tolower($6), ai); print substr(tolower($6), RSTART, RLENGTH), r[2] } }' \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn

     40 claudebot /tag/linux/page/9/
     20 gptbot /old-pricing/
     12 gptbot /.env
     11 claudebot /.env
      9 gptbot /wp-login.php
      7 claudebot /.git/config

Look at the last four lines. No real AI crawler asks for /.env or /.git/config. Those are files that leak passwords when a site is misconfigured, and the requests came from scanners wearing a bot's name. We see the same thing on our own site: on 13 October 2026 our Cloudflare dashboard for rscomputers-ks.com showed many failed requests from scanners that used bot user agents while probing /.env. A count by user agent alone overstates real AI interest, which is why Report 3 exists.

Visitors sent by AI assistants

# as loguser
awk -F'"' '$4 ~ /chatgpt\.com|perplexity\.ai|claude\.ai|copilot\.microsoft\.com|gemini\.google\.com/ { split($2, r, " "); split($4, h, "/"); print h[3], r[2] }' \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn

     14 chatgpt.com /blog/vps-guide/
      6 www.perplexity.ai /blog/vps-guide/
      5 chatgpt.com /vps-server/
      3 claude.ai /docs/setup/

Report 2: GoAccess with AI categories

GoAccess is a free log analyser that runs in the terminal or writes a single HTML file. It already knows Googlebot and bingbot, but it lumps everything into "Crawlers". The --browsers-file option loads your own list, and the category can be any name you choose, so you can give AI bots their own groups. The format is one token per line, a tab, then the category name:

# as loguser
printf '# AI crawlers, one per line: token<TAB>category\nGPTBot\tAI Training\nClaudeBot\tAI Training\nCCBot\tAI Training\nmeta-externalagent\tAI Training\nOAI-SearchBot\tAI Search\nClaude-SearchBot\tAI Search\nPerplexityBot\tAI Search\nChatGPT-User\tAI User\nClaude-User\tAI User\nPerplexity-User\tAI User\n' > ~/ai-bots.list

goaccess /var/log/nginx/access.log --log-format=COMBINED \
  --browsers-file=$HOME/ai-bots.list -o ~/ai-report.html

The "Browsers" panel in the 701 KB report then showed 369 requests as AI Training, 123 as Crawlers (Google, Bing, Apple), 85 as AI Search, 68 as Firefox (our human visitors) and 40 as AI User. Three things we learned along the way:

• Keep category names short. Our first name, "AI User Fetch", appeared cut to "AI User Fetc" in the report.
• --crawlers-only does not include your custom groups. With it, the report showed only the 123 built-in Crawlers. GoAccess treats your AI categories as browsers.
• To get a report of AI bots only, filter first and pipe the result in. GoAccess reads standard input when you give it - as the file name.

# as loguser
grep -E 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|meta-externalagent' \
  /var/log/nginx/access.log \
  | goaccess - --log-format=COMBINED --browsers-file=$HOME/ai-bots.list -o ~/ai-only.html

That report held 494 requests from 11 unique visitors, and its "Not Found URLs" panel listed the same five 404 paths as the awk report. Copy the HTML file to your laptop with scp and open it in a browser, or serve it behind a password. It contains IP addresses, so do not publish it.

Report 3: catch fake bots

This script reads the log, finds every IP that claims to be a known bot, and checks it against that bot's own IP list. For Google, Bing, Apple and Common Crawl it falls back to forward-confirmed reverse DNS: the IP's DNS name must end in the vendor's domain, and that name must resolve back to the same IP. Google documents exactly this method, and it stops someone from simply setting a fake reverse DNS name on their own IP.

First, a script that downloads the lists into ~/ranges. Paste the whole block into your terminal. Everything between the two EOF lines becomes the file:

# as loguser
cat > ~/update-ranges.sh <<'EOF'
#!/bin/sh
mkdir -p "$HOME/ranges" && cd "$HOME/ranges" || exit 1
get() { curl -fsSL --max-time 20 -o "$1.tmp" "$2" && mv "$1.tmp" "$1" || echo "failed: $2"; }
get gptbot.json          https://openai.com/gptbot.json
get searchbot.json       https://openai.com/searchbot.json
get chatgpt-user.json    https://openai.com/chatgpt-user.json
get claude.json          https://claude.com/crawling/bots.json
get perplexitybot.json   https://www.perplexity.com/perplexitybot.json
get perplexity-user.json https://www.perplexity.com/perplexity-user.json
get googlebot.json       https://developers.google.com/static/crawling/ipranges/common-crawlers.json
get bingbot.json         https://www.bing.com/toolbox/bingbot.json
get applebot.json        https://search.developer.apple.com/applebot.json
get ccbot.json           https://index.commoncrawl.org/ccbot.json
ls -l
EOF

All ten URLs answered with HTTP 200 on 13 October 2026, and all use the same layout: a prefixes array of ipv4Prefix or ipv6Prefix entries. That is what keeps the checker short:

# as loguser
cat > ~/verify-bots.py <<'EOF'
#!/usr/bin/env python3
"""Usage: verify-bots.py /var/log/nginx/access.log [ranges-dir]"""
import ipaddress, json, re, socket, sys
from collections import Counter
from pathlib import Path

# user agent token -> (IP list file, reverse DNS suffixes or None)
BOTS = {
    "GPTBot":           ("gptbot.json", None),
    "OAI-SearchBot":    ("searchbot.json", None),
    "ChatGPT-User":     ("chatgpt-user.json", None),
    "ClaudeBot":        ("claude.json", None),
    "Claude-User":      ("claude.json", None),
    "Claude-SearchBot": ("claude.json", None),
    "PerplexityBot":    ("perplexitybot.json", None),
    "Perplexity-User":  ("perplexity-user.json", None),
    "Googlebot":        ("googlebot.json", (".googlebot.com", ".google.com")),
    "bingbot":          ("bingbot.json", (".search.msn.com",)),
    "Applebot":         ("applebot.json", (".applebot.apple.com",)),
    "CCBot":            ("ccbot.json", (".crawl.commoncrawl.org",)),
}
LINE = re.compile(r'^(\S+) \S+ \S+ \[[^\]]+\] "[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')

log = sys.argv[1]
rdir = Path(sys.argv[2] if len(sys.argv) > 2 else Path.home() / "ranges")
nets = {}
for f, _ in set(BOTS.values()):
    data = json.loads((rdir / f).read_text())
    nets[f] = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
               for p in data["prefixes"]]

def fcrdns(ip, suffixes):
    """Reverse DNS must end in the vendor domain and resolve back to the same IP."""
    try:
        host = socket.gethostbyaddr(ip)[0]
        return host.endswith(suffixes) and ip in socket.gethostbyname_ex(host)[2]
    except OSError:
        return False

seen = Counter()
for line in open(log, errors="replace"):
    m = LINE.match(line)
    if not m:
        continue
    ip, status, ua = m.groups()
    for token in BOTS:
        if token.lower() in ua.lower():
            seen[(token, ip)] += 1
            break

cache = {}
for (token, ip), hits in sorted(seen.items()):
    f, suffixes = BOTS[token]
    addr = ipaddress.ip_address(ip)
    ok = any(addr in n for n in nets[f])
    how = "ip-list"
    if not ok and suffixes:
        ok = cache.setdefault(ip, fcrdns(ip, suffixes))
        how = "rdns"
    print(f"{'OK  ' if ok else 'FAKE'} {token:17} {ip:16} {hits:4} hits  ({how})")
EOF

Make both executable and run them:

# as loguser
chmod +x ~/update-ranges.sh ~/verify-bots.py
~/update-ranges.sh
~/verify-bots.py /var/log/nginx/access.log

OK   Applebot          17.241.208.170     12 hits  (ip-list)
OK   CCBot             18.97.14.84        15 hits  (ip-list)
OK   ChatGPT-User      104.210.139.200    24 hits  (ip-list)
OK   Claude-SearchBot  216.73.216.20      14 hits  (ip-list)
OK   Claude-User       216.73.216.30       9 hits  (ip-list)
FAKE ClaudeBot         198.51.100.23      18 hits  (ip-list)
OK   ClaudeBot         216.73.216.10     165 hits  (ip-list)
FAKE GPTBot            198.51.100.7       21 hits  (ip-list)
OK   GPTBot            20.171.207.10     130 hits  (ip-list)
FAKE Googlebot         203.0.113.50        6 hits  (rdns)
OK   Googlebot         66.249.66.1        80 hits  (ip-list)
OK   OAI-SearchBot     104.210.140.130    37 hits  (ip-list)
OK   Perplexity-User   44.208.221.197      7 hits  (ip-list)
OK   PerplexityBot     107.20.236.150     30 hits  (ip-list)
FAKE PerplexityBot     192.0.2.80          4 hits  (ip-list)
OK   bingbot           157.55.39.10       25 hits  (ip-list)

Every fake was caught, and they were exactly the IPs that asked for /.env, /wp-login.php and /.git/config. The check is strict per bot, which is what you want. In our first run we had sent Perplexity-User traffic from a PerplexityBot address by mistake, and the script marked it FAKE, because Perplexity publishes a separate list for each bot.

You can also check one address by hand. Reverse DNS works for Google, Bing and Apple, but not for OpenAI or Anthropic: in our test, 20.171.207.10 (inside GPTBot's range) had no reverse DNS name at all, so for those operators only the IP list counts.

# as loguser
host 66.249.66.1
host crawl-66-249-66-1.googlebot.com
host 20.171.207.10

1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
Host 10.207.171.20.in-addr.arpa. not found: 3(NXDOMAIN)

The lists change, so refresh them every night. This adds one line to the user's crontab:

# as loguser
(crontab -l 2>/dev/null; echo '15 4 * * * $HOME/update-ranges.sh >/dev/null') | crontab -
crontab -l

It prints 15 4 * * * $HOME/update-ranges.sh >/dev/null. Once you know which IPs are fakes, you can block them with a firewall or put a proof-of-work gate in front of the site. Our guide to blocking AI scrapers with Anubis covers the second option.

How fast is it on a real-sized log?

We repeated the 685 lines until the log had 1,000,100 lines and 178 MB, and timed each tool inside the same 1 GiB container.

The custom browsers list cost GoAccess nothing measurable: 14.24 seconds with it, 14.23 without. Real logs have far more unique IPs and URLs than our replay, so GoAccess will use more memory on them, but none of this needs more than the smallest server. The time to plan for is disk space if you keep months of logs.

Using the numbers for SEO and AI answers

AEO (answer engine optimisation) means making your pages easy for AI assistants to find, read and cite. Your log is the only place that shows what those assistants actually do on your site, and it answers four practical questions.

Which pages does AI read?

Compare the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) with your list of important pages. A product or pricing page that no AI search bot has fetched in a month cannot be quoted in an answer. Check internal links to it, the sitemap, and whether robots.txt or a firewall rule blocks those bots by accident. If a training crawler reads an archive page 40 times a day while your guides get nothing, as ClaudeBot did with /tag/linux/page/9/ in our lab, consider a noindex or a Disallow for those archive paths.

Which 404s should you fix?

A 404 that a real AI bot keeps requesting usually means an old URL still lives in its index or in someone's citation. In our lab GPTBot asked for /old-pricing/ 20 times. Redirect such URLs with a 301 to the current page. Ignore 404s that come from fake bots, because those are scanners.

Do the assistants send visitors?

A visit with chatgpt.com, perplexity.ai or claude.ai in the Referer is a human who clicked a citation. OpenAI says ChatGPT also adds utm_source=chatgpt.com to links in its search answers (OpenAI publisher FAQ), so look for that in the request path too. These counts are a floor, not the full number: Cloudflare found in July 2025 that traffic from Claude's native app carries no Referer, and it expects other native apps to behave the same way.

Are user fetches growing?

ChatGPT-User, Claude-User and Perplexity-User requests mean someone asked about your page in a chat. Track them week by week. If they grow while referrals stay flat, people are reading your content inside the assistant. If you want to serve those agents a lighter version of each page, see serving Markdown to AI agents with Nginx or Caddy.

For alerts and long-term graphs, the same numbers can go into Prometheus and Grafana; our Uptime Kuma and Grafana guide sets up the stack. And if you run your own crawler and want to be on the right side of all this, read our guide to running web scrapers and crawlers on a VPS.

Frequently asked questions

How do I know if ChatGPT is crawling my website?

Search your web server's access log for GPTBot, OAI-SearchBot and ChatGPT-User. GPTBot collects training data, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User fetches a page when a user asks about it. Then check the IP against OpenAI's published lists, because the name alone can be faked.

Why don't AI bots show up in Google Analytics?

Google Analytics, Plausible and similar tools count a visit only when the browser runs their JavaScript tag. AI crawlers download the HTML and never run that script, so they are invisible to tag-based analytics. The server access log records every request, which is why log analysis is the only way to see them.

How can I tell a real ClaudeBot from a fake one?

Check whether the request's IP address is inside one of the ranges in Anthropic's file at claude.com/crawling/bots.json, which covers ClaudeBot, Claude-User and Claude-SearchBot. Reverse DNS does not help here: Anthropic does not document it, and the Anthropic address we looked up in our lab had no reverse DNS name. A request that names ClaudeBot from any other IP is not from Anthropic.

What is the difference between GPTBot and ChatGPT-User?

GPTBot crawls on its own schedule to collect content that may train OpenAI's models, and it obeys robots.txt. ChatGPT-User visits a single page because a person in ChatGPT asked for it, and OpenAI says robots.txt rules may not apply to it. Blocking GPTBot does not remove you from ChatGPT search, because OAI-SearchBot controls that.

Does Google-Extended crawl my site?

No. Google-Extended is only a robots.txt token. Google's own crawlers fetch the pages, and the token tells Google whether that content may be used to train Gemini and ground its answers. It has no user agent of its own, so it never appears in an access log.

Can GoAccess detect AI crawlers?

Yes, with a custom list. GoAccess 1.9.3 accepts --browsers-file with lines of a user agent token, a tab and a category, so GPTBot or ClaudeBot can be grouped as "AI Training" or "AI Search". Custom categories are not counted by --crawlers-only, so filter the log with grep first if you want an AI-only report.

Run it on your own server

All three reports run comfortably on the smallest VPS: in our lab, a 1 GiB container processed a million log lines in seconds with under 16 MB of RAM. If you host your site with us, the access log is already on the server, so you can start with the awk lines today. Pick a VPS in Amsterdam, Dublin or Prishtina on the plans page, where availability by city is listed, or message us on Telegram if you want this set up for you; setup work is quoted per job.

← All articles