Block AI Scrapers on Your Own Server: Anubis on a VPS (Tested)

To block AI scrapers on your own server, put Anubis in front of your website. Anubis is a small, free, open-source program (MIT licence, written in Go) that sits between Nginx and your site. It sends known AI crawlers such as GPTBot an error page and asks every browser to solve a short proof-of-work puzzle in JavaScript. A real visitor waits about half a second once a week, while a scraper pays for every new session. We installed Anubis 1.28.1 on Debian 13 behind Nginx, threw curl, a fake GPTBot, a fake Googlebot and a headless Chrome at it, and measured what it costs in RAM and CPU. Everything below comes from that lab.

Key facts, checked on 13 October 2026:

• The latest Anubis release is v1.28.1, published on 9 October 2026. It reverts a systemd isolation change from the 1.28.0 pre-release that failed in production, so skip 1.28.0.
• The GitHub repository had 23,322 stars. The project started in March 2025.
• Cloudflare reported on 1 July 2026 that AI training made up 52% of crawler requests in June 2026, up from 22% in spring 2025 (Cloudflare blog).
• Fewer than 1% of sites on Cloudflare block search bots, and 17% use some way to block AI training (Cloudflare, 15 September 2026).
• In our lab Anubis used about 48 MB of RAM when idle and peaked at 70 MB while it served 20,000 challenge pages at 2,413 requests per second.

What Anubis actually does to a request

Anubis is a reverse proxy, which means it receives the request first and only passes it on to your real website if it is happy. For every request it walks through a list of rules from top to bottom, and the first rule that matches decides:

• ALLOW sends the request straight to your site. The default list allows Googlebot, Bingbot, Applebot, DuckDuckBot and several other search crawlers, but only when the request comes from that company's published IP ranges.
• DENY returns the "Oh noes!" error page. The default policy denies GPTBot, ClaudeBot, Bytespider, PerplexityBot, headless Chrome and a long list of other AI agents.
• CHALLENGE returns a page with a puzzle. The browser has to find a number that, hashed with SHA-256 together with a random string, starts with a set count of zeros. Then it gets a signed cookie and is waved through.
• WEIGH adds or removes points. Anything whose user agent contains "Mozilla" gets 10 points, and a score between 10 and 19 earns an easy puzzle (difficulty 2).

The part most guides skip: anything that matches no rule is allowed. The default allow behaviour is deliberate, so git clients, RSS readers and API tools keep working. It also means a plain curl or Python script that does not pretend to be a browser walks right in. We show the fix further down.

The cookie is called techaro.lol-anubis-auth followed by a short hash. In our test it was valid for 7 days, which matches the COOKIE_EXPIRATION_TIME default of 168 hours.

The lab

We ran everything in a Debian 13 container limited to 1 GB of RAM and 2 vCPUs, on one of our Amsterdam test servers. That is the same memory as our smallest plan. The setup has three parts:

1. Nginx listens on ports 80 and 443 and terminates TLS (it turns HTTPS into plain HTTP).
2. Nginx passes every request to Anubis on 127.0.0.1:8923.
3. Anubis passes the requests it likes to the real website, a second Nginx server block on 127.0.0.1:3000.

If you have never set up a reverse proxy or a certificate, read our reverse proxy and SSL guide first, and lock down the server with our Debian 13 hardening checklist.

Step 1: install Nginx and the Anubis package

Anubis publishes .deb, .rpm and tarball packages on every GitHub release, plus a Docker image at ghcr.io/techarohq/anubis. On Debian the .deb is the simplest route because it ships a systemd unit and all the rule files.

# as root
apt-get update
apt-get install -y nginx curl ca-certificates
cd /tmp
curl -fsSLO https://github.com/TecharoHQ/anubis/releases/download/v1.28.1/anubis_1.28.1_amd64.deb
apt-get install -y ./anubis_1.28.1_amd64.deb
anubis --version

What we saw: the package is 24 MB, Nginx is 1.26.3, and the last line prints Anubis v1.28.1. The package installs /usr/bin/anubis (45 MB, one static binary), a template unit anubis@.service, the sample /etc/anubis/default.env and the ready-made rules under /usr/share/doc/anubis/data/. On an ARM server, download anubis_1.28.1_arm64.deb instead.

Step 2: point Nginx at Anubis

Put your real site on a local port and send the public side through Anubis. In the lab we used a self-signed certificate for the made-up name demo.test. On a real site, point the two ssl_certificate lines at your Let's Encrypt files and add the usual redirect from port 80 to 443.

# as root
mkdir -p /var/www/demo
echo '<!doctype html><title>Demo site</title><h1>Hello from the backend</h1>' > /var/www/demo/index.html
openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj /CN=demo.test \
  -addext subjectAltName=DNS:demo.test \
  -keyout /etc/ssl/private/demo.key -out /etc/ssl/certs/demo.crt
echo "127.0.0.1 demo.test" >> /etc/hosts

cat > /etc/nginx/sites-available/demo <<'EOF'
# Backend: the real website, only reachable from this server
server {
    listen 127.0.0.1:3000;
    server_name _;
    root /var/www/demo;
    index index.html;
    set_real_ip_from 127.0.0.1;
    real_ip_header X-Real-IP;
}

upstream anubis {
    server 127.0.0.1:8923;
}

# Public side: everything goes through Anubis first
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    ssl_certificate /etc/ssl/certs/demo.crt;
    ssl_certificate_key /etc/ssl/private/demo.key;
    server_name _;
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://anubis;
    }
}
EOF
rm -f /etc/nginx/sites-enabled/default
ln -s /etc/nginx/sites-available/demo /etc/nginx/sites-enabled/demo
nginx -t && systemctl reload nginx

Expected output: nginx: configuration file /etc/nginx/nginx.conf test is successful. The X-Real-IP header matters. Anubis checks the visitor's IP against Google's and Bing's address lists, so if Nginx does not pass the real address, every visitor looks like 127.0.0.1 and the search engine allow rules never match.

Step 3: configure and start Anubis

The unit file is a template. anubis@demo.service reads /etc/anubis/demo.env, so you can run one Anubis per site.

# as root
cat > /etc/anubis/demo.env <<'EOF'
BIND=127.0.0.1:8923
DIFFICULTY=4
METRICS_BIND=127.0.0.1:9090
SERVE_ROBOTS_TXT=0
TARGET=http://127.0.0.1:3000
POLICY_FNAME=/etc/anubis/demo.botPolicies.yaml
EOF
cp /usr/share/doc/anubis/botPolicies.yaml /etc/anubis/demo.botPolicies.yaml
systemctl enable --now anubis@demo.service
systemctl is-active anubis@demo

It prints active. The log (journalctl -u anubis@demo) shows a line with "msg":"listening" and "version":"v1.28.1", plus some warnings that are safe to read once:

• No GeoIP database is configured, so the two sample rules that add points for some countries and ASNs do nothing. They need a free MaxMind account, which we did not set up.
• "generating random key" means the signing key is new on every restart, so all visitors solve the puzzle again after a restart. Set ED25519_PRIVATE_KEY_HEX if you run more than one Anubis behind a load balancer.
• REDIRECT_DOMAINS is not set. Set it to your own domain names so Anubis only redirects to them after a challenge.

Plain HTTP is where people get stuck. By default the auth cookie is marked Secure (COOKIE_SECURE=true in the installation docs), and browsers refuse to store a Secure cookie from an HTTP page. We tried HTTP first with COOKIE_SECURE=false. The cookie still arrived with SameSite=None and Partitioned and the browser still dropped it. Headless Chrome solved the puzzle in 55 ms, Anubis logged "user has cookies disabled", and the browser looped forever. Over HTTPS, with the cookie settings back at their defaults, the same browser got through at once. Run Anubis behind HTTPS.

Step 4: test it with curl, a fake GPTBot and a fake Googlebot

These checks only read the first response, so plain HTTP on the server itself is enough. No cookie is involved yet.

# as root, on the server
curl -s -A "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0" http://127.0.0.1/ | grep -o "<title>.*</title>"
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" http://127.0.0.1/ | grep -o "<title>.*</title>"
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.1" http://127.0.0.1/

Output, one line each:

# output
<title>Making sure you&#39;re not a bot!</title>
<title>Oh noes!</title>
200

The Firefox user agent gets the puzzle page. GPTBot gets the error page with the code 26301432c34cf028, which the log maps to the rule ai-crawlers-training. Notice the status code: 200, not 403. That is on purpose. The default policy sets status_codes for CHALLENGE and DENY to 200 because, as the comment in the file says, aggressive scrapers seem to stop once they get a 200. If you count blocks in your access logs by status code, you will count nothing. Use the metrics endpoint instead (Step 7), or read our guide to AI crawler log analysis.

Now the search engine check. A request that claims to be Googlebot from the wrong address gets the puzzle, which Googlebot does not solve. To prove the IP rule works, we sent requests straight to Anubis with a forged X-Real-IP, once with an address from Google's published crawler ranges and once with a documentation address. The shipped rule starts like this:

# /usr/share/doc/anubis/data/crawlers/googlebot.yaml (first lines)
- name: googlebot
  user_agent_regex: \+http\://www\.google\.com/bot\.html
  action: ALLOW
  remote_addresses:
    - 2001:4860:4801:10::/64
    - 2001:4860:4801:11::/64

# as root, on the server
GUA="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
curl -s -H "X-Real-IP: 66.249.66.1" -A "$GUA" http://127.0.0.1:8923/ | head -c 80; echo
curl -s -H "X-Real-IP: 203.0.113.50" -A "$GUA" http://127.0.0.1:8923/ | grep -o "<title>.*</title>"

# output
<!doctype html><title>Demo site</title><h1>Hello from the backend</h1>
<title>Making sure you&#39;re not a bot!</title>

The Googlebot rule ships with more than 300 IPv4 and IPv6 ranges, taken from the list Google publishes for verifying Googlebot. The same pattern covers Bingbot, Applebot and the others. This test only works because port 8923 listens on 127.0.0.1. Never expose Anubis directly to the internet, or anyone can forge the header.

Step 5: close the curl gap and protect what must stay open

Out of the box, a plain curl request got the real page, because it sends no "Mozilla" and matches no rule. Most cheap scrapers send a Python or Go user agent. We added our own rules in a separate file and imported it at the top of the policy:

# as root
cat > /etc/anubis/my-rules.yaml <<'EOF'
# Keep robots.txt, favicon and /.well-known open for everyone
- import: (data)/common/keep-internet-working.yaml

# Let my own uptime monitor through without a challenge
- name: my-uptime-monitor
  remote_addresses:
    - 10.141.188.1/32
  action: ALLOW

# Command-line tools do not send Mozilla, so by default they pass.
# Send them the challenge too (they cannot solve it).
- name: cli-clients
  user_agent_regex: (?i)^(curl|wget|python-requests|python-httpx|go-http-client|scrapy)
  action: CHALLENGE
EOF
sed -i 's|^bots:$|bots:\n  - import: /etc/anubis/my-rules.yaml|' /etc/anubis/demo.botPolicies.yaml
systemctl restart anubis@demo

Replace 10.141.188.1/32 with the IP of your own monitoring, for example an Uptime Kuma server. After the restart, curl got the puzzle page (difficulty 4, from DIFFICULTY=4) or the error page, and a request from the monitor's address got the real site.

Our first version of this file did not have the keep-internet-working import at the top. The result: curl asking for /robots.txt got a puzzle instead of the file, because our curl rule matched first. A crawler that cannot read robots.txt cannot obey it either, so keep that import above any rule that challenges non-browsers. If your site has an API, webhooks or RSS feeds, add ALLOW rules for those paths in the same place. The package ships examples in /usr/share/doc/anubis/data/apps/ and /usr/share/doc/anubis/data/common/.

Turn your robots.txt into rules

The package includes a small converter. We fed it a robots.txt that blocks GPTBot completely and keeps everyone out of /admin/:

# as root
printf "User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nDisallow: /admin/\n" > /tmp/robots.txt
anubis-robots2policy -input /tmp/robots.txt

# output
- action: DENY
  expression: userAgent.contains("GPTBot")
  name: robots-txt-policy-blacklist-1
- action: CHALLENGE
  expression: path.startsWith("/admin/")
  name: robots-txt-policy-disallow-2

How long the puzzle takes, and what Anubis costs the server

We drove Chromium 155 headless through its DevTools protocol with a short Python script, as a normal user, and timed how long it took from the first request to the real page. Each run started with a fresh profile, so no cookie. All numbers are from our lab container with 2 shared vCPUs.

Two lessons. First, each extra level multiplies the average work by 16, and luck matters a lot: at difficulty 5 one run finished in 1.3 seconds and another took almost 16. Difficulty 4 is about the most you should ask from people on older phones. Second, a headless browser that lies about its user agent got through in half a second. Anubis does not stop a determined scraper. It makes each fresh session cost CPU time, so a crawler that rotates thousands of IPs and drops cookies pays much more than a person who visits once a week.

For the server side we used ab (ApacheBench) with 50 parallel connections:

The whole container used 101 MB of its 1 GB afterwards. About half a millisecond of CPU per challenge is cheap next to a WordPress page that hits a database. If your server is already struggling with crawlers, Anubis usually lowers the total load, because the expensive pages stop being rendered for bots. To check whether bots are really your problem, see why your VPS is slow. Anubis is not a DDoS filter, though. A flood that fills your port never reaches it, so read how to detect a DDoS on a Linux server for that.

Watch it work

# as root, on the server
curl -s http://127.0.0.1:9090/metrics | grep -E "^anubis_(challenges_issued|policy_results)"

After the load test this showed anubis_challenges_issued at 20,006 and anubis_policy_results{action="DENY",...,rule="bot/ai-crawlers-training"} at 20,000. Keep the metrics port on 127.0.0.1.

Anubis, Cloudflare or robots.txt?

Start with robots.txt anyway. Some opt-outs only exist there: Google's AI training opt-out is the Google-Extended token, which is a robots.txt name, not a separate crawler, and the Anubis rule file itself warns that some training opt-outs only work through robots.txt. Our article on content signals, robots.txt and llms.txt covers the new options. Add Anubis when crawlers ignore it and your server feels the load, which is typical for git forges, wikis, forums and shops with many filter URLs.

Cloudflare moved in the same direction this year. On 1 July 2026 it replaced its single "Block AI bots" switch with separate Search, Training and Agent settings, available on every plan including Free (Cloudflare changelog), and its AI Crawl Control dashboard shows which crawlers visit (changelog, February 2026). Since 15 September 2026, new domains that say they carry ads get Training set to "Disallow AI Training" and Agents blocked on pages with ads, while Search stays allowed (Cloudflare blog). If you would rather charge crawlers than block them, see pay per crawl and HTTP 402.

The SEO risk, and how to avoid it

Anubis can hurt your rankings if you change the defaults carelessly. The challenge page carries <meta name="robots" content="noindex,nofollow">, so a search crawler that gets the puzzle instead of your page sees a page that says "do not index me". Keep these points in mind:

• Keep the (data)/crawlers/_allow-good.yaml import. It is what lets Googlebot, Bingbot and others in, by IP.
• Never add a catch-all rule like (?i:bot|crawler) above it. The default policy has that rule commented out with a warning about false positives.
• The default "aggressive" AI block also denies GoogleOther, PerplexityBot and ChatGPT's user fetches. If you want to be cited by AI search, switch to (data)/meta/ai-block-moderate.yaml. That file still blocks training crawlers but allows documented search and user-triggered fetchers by their IP lists.
• After you deploy, run the URL Inspection live test in Google Search Console and check that it renders your page, not the puzzle.
• Link previews in chat apps may fail. The package has rules for some of them in data/clients/.

Other downsides to accept: visitors need JavaScript (the no-JS page says so), some privacy add-ons such as JShelter break the puzzle, and the warning page has its own look, which you can only change with the paid BotStopper edition.

Frequently asked questions

Does Anubis block Googlebot?

No, not with the default policy. Anubis 1.28.1 allows Googlebot, Bingbot, Applebot, DuckDuckBot and several other search crawlers when the request comes from their published IP ranges. A fake Googlebot from another address gets the challenge. In our lab, a Googlebot user agent from 66.249.66.1 reached the page, and the same user agent from 203.0.113.50 got the puzzle.

How much RAM does Anubis need?

Very little. In our lab Anubis 1.28.1 used 48 MB when idle and peaked at 71 MB while it answered 20,000 requests at over 2,400 per second on 2 vCPUs. A 1 GB VPS has room for Anubis, Nginx and a small website.

Can AI scrapers solve the Anubis challenge?

Yes, a real browser can. In our test a headless Chrome with a normal user agent passed the default difficulty in about half a second. Anubis works by making each new session cost CPU time and by denying known AI user agents outright, which is enough to stop most bulk scrapers, but it is not an absolute wall.

Is Anubis free to use?

Yes. Anubis is open source under the MIT licence and you can run it on any server. Techaro also sells BotStopper, a commercial edition with extras such as custom branding and headless browser detection.

Why does Anubis return HTTP 200 when it blocks a bot?

The default policy sets the status code for both challenge and deny pages to 200, because the developers found that aggressive scrapers stop retrying once they get a 200. You can change this under status_codes in the policy file. Count blocks with the metrics endpoint, not with status codes in your access log.

Do I still need robots.txt if I run Anubis?

Yes. Honest crawlers read robots.txt, and some opt-outs, such as Google-Extended for Google's AI training, only exist there. Anubis enforces rules on crawlers that ignore robots.txt. Keep robots.txt reachable for every client by putting the keep-internet-working rules first.

Where to run it

Anubis is one binary, one config file and one systemd unit, so it fits any of our plans. A VPS Nano (1 vCPU, 1 GB RAM) is enough for Anubis in front of a small static site. For WordPress or a Forgejo server that crawlers like to hammer, a VPS Mini (4 vCPU, 4 GB) or a VDS Small with 4 dedicated vCPUs gives the backend room to breathe. All plans have NVMe storage, unmetered traffic and their own IPv4 and IPv6, in Amsterdam, Dublin and Prishtina. Availability by city is on the plans page. Pick a server on the VPS and VDS plans page, or ask us on Telegram if you want help sizing it or a quote for setting it up.

← All articles