HTTP 402 and Pay Per Crawl Explained: Can You Charge AI Bots From Your Own Server?

Yes, you can charge AI bots from your own server in the technical sense: Nginx can answer GPTBot or ClaudeBot with HTTP 402 Payment Required and a price header in about twenty lines of configuration, while browsers and Googlebot still get the page. What you cannot do on your own is get paid. HTTP 402 has no built-in payment system, and as of 13 October 2026 no AI company publicly pays a price that a site invents for itself. Money only moves when both sides join the same marketplace, such as Cloudflare's Pay Per Crawl and Pay Per Use, or the x402 payment protocol. This article explains how those systems work, what changed in 2026, who would actually pay a small site, and the Nginx setup we tested in our lab.

Key facts, checked on 13 October 2026:

• HTTP 402 has existed since the early web, but the HTTP standard still calls it "reserved for future use" (RFC 9110). Browsers and crawlers do nothing special with it unless they are built to.
• Cloudflare launched Pay Per Crawl on 1 July 2025. A crawler that asks without paying gets a 402 with a crawler-price header (Cloudflare blog). The feature is still in closed beta, and Cloudflare is the merchant of record (Cloudflare docs).
• On 1 July 2026 Cloudflare announced it was "evolving Pay Per Crawl into Pay Per Use", with Ceramic.ai and You.com as the first partners (press release). Pay Per Use entered beta on 30 September 2026 (Cloudflare blog).
• x402, the open payment protocol Coinbase started, moved to the Linux Foundation. The x402 Foundation launched operationally on 14 July 2026 with 40 member organisations (Linux Foundation).
• In July 2025, Anthropic's crawlers fetched about 38,066 pages for every visitor they sent back, and OpenAI's about 1,091. Google's ratio was 5.4 (Cloudflare, August 2025).

HTTP 402 in one paragraph

Every web response carries a three-digit status code. 200 means "here is the page", 403 means "you may not have it", 404 means "it does not exist". 402 means "you can have it if you pay". The code was reserved in the 1990s for a payment system that never arrived, so for thirty years almost nothing used it. AI crawlers gave it a job: a 402 is a polite middle ground between letting a bot take everything for free and blocking it outright. The catch is that the code only says "pay". It does not say how, to whom, or in what currency. Pay Per Crawl, Pay Per Use and x402 are three different answers to that missing part.

From Pay Per Crawl to Pay Per Use: the dated timeline

One detail matters for anyone reading older coverage. Cloudflare's July press release used the word "evolving", but the September Pay Per Use post describes the two as separate options: Pay Per Crawl "charges for access", Pay Per Use pays for what happens next, and "publishers can choose the model that suits them". Pay Per Crawl's own documentation, last updated on 28 July 2026, still lists it as a closed beta. So in October 2026 both exist, and neither is open to everyone.

How a Pay Per Crawl request looks

The 2025 launch post spells out the headers. A crawler that sends no payment intent gets HTTP 402 with crawler-price: USD 0.01 (or whatever the site set). It can then retry with crawler-exact-price to accept that price, or send crawler-max-price up front, in which case the site answers 200 and confirms the charge in a crawler-charged header. Payment intent only counts from a crawler that signs its requests with Web Bot Auth and is registered with Cloudflare. That is why a header alone, sent from your own Nginx, cannot collect anything: there is no registered buyer behind it and nobody to send the invoice to.

x402 and Web Bot Auth: the wallet and the ID card

x402 is an open protocol for payments inside HTTP. The server replies 402 with a PAYMENT-REQUIRED header describing the price and where to pay. The client signs a payment and repeats the request with a PAYMENT-SIGNATURE header, and the server confirms with PAYMENT-RESPONSE (x402 specification). A "facilitator" service checks and settles the payment so your server does not need to run blockchain software. Today the payments are mostly USDC stablecoin on networks such as Base; Cloudflare's Monetization Gateway settles in USDC on Base through Coinbase's facilitator. Card and bank payments are on the foundation's list, not yet the normal case.

x402 works from your own server. The open-source SDKs include middleware for Node.js and other stacks, so a self-hosted site can sell a page or an API call to any client that carries a wallet. The question is who carries one. A developer who set up an x402 paywall on his own site wrote on 15 September 2026 that "all the payments so far have come from my own agents", and that GPTBot and ClaudeBot only showed up in his tests as spoofed user agents (Suganthan Mohanadasan). The big training crawlers do not pay x402 invoices.

Web Bot Auth answers a different question: is this request really from the bot it claims to be? The bot signs each request with a private key (HTTP Message Signatures, RFC 9421) and adds a Signature-Agent header pointing to a directory of its public keys. OpenAI says ChatGPT agent signs every outbound request this way, with Signature-Agent set to "https://chatgpt.com" (OpenAI help). Cloudflare now counts such "signed agents" as verified bots (Cloudflare docs). For billing this is the foundation: you cannot invoice a bot whose identity anyone can fake with one curl flag.

Who would actually pay a small website?

Most guides skip this part. A small site can realistically expect money from three places, and none of them is a self-made 402:

• A marketplace with buyers already signed up. Pay Per Use has its buyers set the price and opt-in publishers accept or decline offers. If your site is on Cloudflare and you are let into the beta, you can earn from Ceramic-style search citations without negotiating anything.
• Agents with a wallet. Developer tools and research agents that pay per call through x402 exist, but they mostly buy APIs, data and tools, not blog posts. Cloudflare's own launch examples are a search API, market signals and a PDF API.
• A direct licence. Cloudflare counted more than 50 major content licensing deals between publishers and AI platforms in the year before July 2026. Those went to large publishers with archives worth negotiating over.

A personal blog, a shop or a local business site has little to sell to a training crawler, and the same pages are usually available elsewhere. A 402 from such a site is, in practice, a block with a friendly message. That can still be the right call. Just do not budget the income.

Cited or paid: the visibility trade-off

AI companies run different bots for different jobs, and the split matters more than the price. OpenAI separates GPTBot (training), OAI-SearchBot (search results in ChatGPT) and ChatGPT-User (a person asked ChatGPT to open your page) on its bots page. Anthropic separates ClaudeBot, Claude-SearchBot and Claude-User in the same way (Anthropic help).

That is why the config below prices only training crawlers. It is also a choice, not a rule. RS Computers lets every AI crawler read this site, training bots included, because being quoted correctly by ChatGPT or Claude when someone asks about a VPS in Amsterdam, Dublin or Prishtina is worth more to us than a few cents per page. If you only want to state your preferences instead of enforcing them, read our guide to content signals, robots.txt and llms.txt. If bots ignore your robots.txt and hammer the server, a proof-of-work wall like Anubis is the stronger tool.

Tested: a 402 with a price, straight from Nginx

We built this on a Debian 13 container with 2 vCPUs and a 512 MiB memory limit, on a KVM server in our Amsterdam lab. Nginx 1.26.3 from Debian installed in 6 seconds, and the whole container used 21 MB of RAM afterwards. Nothing here needs Cloudflare or an account anywhere.

Step 1: install Nginx

# as root
apt-get update
apt-get install -y nginx curl apache2-utils
nginx -v

Expected output: nginx version: nginx/1.26.3. The apache2-utils package is only there for the load test at the end.

Step 2: list the bots and their price

A map turns the User-Agent of each request into a price. An empty price means free. Save this as /etc/nginx/conf.d/ai-price.conf:

# as root: /etc/nginx/conf.d/ai-price.conf
# Price per page for AI training crawlers, matched by User-Agent.
# An empty value means "free": search engines and browsers get the page.
map $http_user_agent $ai_price {
    default                 "";
    ~*GPTBot                "USD 0.01";
    ~*ClaudeBot             "USD 0.01";
    ~*CCBot                 "USD 0.01";
    ~*Bytespider            "USD 0.01";
    ~*Meta-ExternalAgent    "USD 0.01";
}

# Separate log for every priced request, so you can see who would have paid.
log_format ai_price '$time_iso8601 $remote_addr $status "$request" '
                    '"$http_user_agent" price="$ai_price"';

Step 3: answer 402 with a crawler-price header

Replace the server block in /etc/nginx/sites-available/default. We use the same header name as Cloudflare, so a crawler that already understands Pay Per Crawl reads the price the same way. robots.txt and the sitemap stay free for every bot, which is also how Cloudflare does it.

# as root: /etc/nginx/sites-available/default
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    root /var/www/html;
    index index.html;

    access_log /var/log/nginx/access.log;
    access_log /var/log/nginx/ai-price.log ai_price if=$ai_price;

    # These files stay free for every bot, as on Cloudflare.
    location = /robots.txt  { try_files $uri =404; }
    location = /sitemap.xml { try_files $uri =404; }

    location / {
        if ($ai_price) {
            return 402;
        }
        try_files $uri $uri/ =404;
    }

    error_page 402 /402.json;
    location = /402.json {
        internal;
        default_type application/json;
        add_header crawler-price $ai_price always;
        add_header Cache-Control "no-store" always;
        return 402 '{"error":"payment required","price":"$ai_price","contact":"licensing@example.com"}\n';
    }
}

Two details are easy to miss. Nginx drops add_header on error responses unless you add always. And Cache-Control: no-store stops a cache or CDN in front of you from serving the 402 to normal visitors. Put your real licensing address in the JSON body, because a person reading logs at an AI company is the only one who might ever reply.

# as root
nginx -t && systemctl reload nginx

Expected output: syntax is ok and test is successful. Wait a second before testing: in our lab, a curl sent in the same instant as the reload still reached the old worker processes and got the old answer.

Step 4: test it as a normal user

# as a normal user (we used "tester")
curl -si -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)" http://127.0.0.1/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" http://127.0.0.1/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" http://127.0.0.1/

What we saw for GPTBot:

# output of the first command (Date and Connection lines left out)
HTTP/1.1 402 Payment Required
Server: nginx
Content-Type: application/json
Content-Length: 82
crawler-price: USD 0.01
Cache-Control: no-store

{"error":"payment required","price":"USD 0.01","contact":"licensing@example.com"}

The Chrome and Googlebot requests printed 200. ClaudeBot got 402, OAI-SearchBot got 200, GPTBot asking for /robots.txt got 200, and GPTBot asking for a page that does not exist got 402 rather than 404, so the bot learns nothing about your URL structure without paying. A HEAD request also returned 402.

Step 5: see what you would have earned

# as root
grep -o 'GPTBot\|ClaudeBot\|CCBot\|Bytespider\|Meta-ExternalAgent' /var/log/nginx/ai-price.log | sort | uniq -c | sort -rn
awk '$3==402' /var/log/nginx/ai-price.log | wc -l | awk '{printf "%d priced hits = USD %.2f you were not paid\n", $1, $1*0.01}'

After our load test the lab log showed 20007 GPTBot and 1 ClaudeBot, and the second line printed 20007 priced hits = USD 200.07 you were not paid. That last word is the point. The log is a useful negotiating number if a buyer ever asks, but no invoice goes out.

What it costs the server

A 402 is cheaper to serve than a real page. With ApacheBench sending 20,000 requests, 50 at a time, from inside the same 2-vCPU container, Nginx answered GPTBot with 402 at 15,398 requests per second and a normal browser with the 200 page at 15,732. Both runs had 0 failed requests, and each Nginx worker used about 6 MB of RAM. Even the smallest VPS Nano has far more headroom than a crawler will ever use.

# as a normal user
ab -n 20000 -c 50 -H "User-Agent: GPTBot/1.3" http://127.0.0.1/

Verify the bot before you price it, or before you let it in free

Charging a fake GPTBot does no harm: a scraper pretending to be GPTBot just gets the 402 too. The danger is on the other side. Anyone can type "Googlebot" into a User-Agent, and the free lane you keep for search engines becomes the scraper's way in. Google publishes the IP ranges of its crawlers, so Nginx can check them. The old googlebot.json address now redirects to a combined list, which we used:

# as root
apt-get install -y jq
curl -s https://developers.google.com/static/crawling/ipranges/common-crawlers.json -o /tmp/googlebot.json
jq -r .creationTime /tmp/googlebot.json
jq -r '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) + " 1;"' /tmp/googlebot.json > /etc/nginx/googlebot-ips.conf
wc -l /etc/nginx/googlebot-ips.conf

On 13 October 2026 the list was created on 2026-10-12T14:46:34 and held 317 prefixes. Then add a second file, /etc/nginx/conf.d/verify-google.conf:

# as root: /etc/nginx/conf.d/verify-google.conf
# 1 when the client IP is in Google's published crawler ranges.
geo $from_google {
    default 0;
    include /etc/nginx/googlebot-ips.conf;
}

# 1 when the User-Agent says Googlebot but the IP is not Google's.
map "$from_google:$http_user_agent" $fake_google {
    default          0;
    "~*^0:.*Googlebot" 1;
}

In the location / block, put this check above the 402 check, then run nginx -t and reload again:

# as root: inside location / in /etc/nginx/sites-available/default
        if ($fake_google) {
            return 403;
        }

In our lab, a Googlebot User-Agent from 127.0.0.1 now got 403, while Chrome still got 200 and GPTBot still got 402. Our lab has no Google address, so to test the allowed path we temporarily added the line 127.0.0.2 1; to the geo block and sent the request with curl --interface 127.0.0.2; that returned 200. Remove such a line on a real server. Refresh the IP list with a weekly cron job, because Google changes it.

The same pattern works for OpenAI, which publishes IP lists for its bots, and for any bot that only gets free access after a deal. Signed requests with Web Bot Auth are the stronger check, but verifying an RFC 9421 signature needs extra code, such as a small service behind Nginx; plain Nginx cannot do it. For working out which bots really hit your site and which of them are fakes, see our self-hosted AI crawler log analysis guide.

Frequently asked questions

What is HTTP 402 Payment Required?

HTTP 402 is a status code that tells the client it must pay to get the resource. RFC 9110 still lists it as reserved for future use, so it has no standard payment method. Cloudflare Pay Per Crawl and the x402 protocol each add their own headers on top of it to carry a price and a payment.

Do AI companies pay when my server returns 402?

Not if you built the 402 yourself. As of October 2026, no AI company publicly pays prices set by independent servers. Payments happen inside marketplaces where the buyer has signed up, such as Cloudflare Pay Per Crawl, Cloudflare Pay Per Use with partners like Ceramic.ai, or agents that carry an x402 wallet.

Is Cloudflare Pay Per Crawl free to join?

You cannot simply switch it on. Pay Per Crawl is a closed beta that you apply for with a form, or through your account executive on Enterprise plans, and Cloudflare acts as the merchant of record. Your site must be proxied through Cloudflare. Pay Per Use, in beta since 30 September 2026, is reached through the Cloudflare dashboard under Monetize.

Will returning 402 to AI bots hurt my Google rankings?

Not if you only target AI training crawlers such as GPTBot or ClaudeBot and keep Googlebot on 200. Returning 402 to AI search bots like OAI-SearchBot or PerplexityBot can remove your pages from AI answers and their source links. Google-Extended is only a robots.txt token, so it cannot be priced separately from Googlebot.

What is the difference between x402 and Pay Per Crawl?

x402 is an open protocol, now run by the x402 Foundation at the Linux Foundation, that lets any server and any client exchange a payment over HTTP, usually in USDC stablecoin. Pay Per Crawl is Cloudflare's product for charging verified AI crawlers, with Cloudflare handling identity and billing. Cloudflare's Monetization Gateway uses x402 underneath.

How do I know a request really comes from GPTBot?

Check the source IP against the ranges the company publishes, or verify a Web Bot Auth signature when the bot sends one. The User-Agent string alone proves nothing, because any client can set it. Reverse DNS works for Googlebot and Bingbot.

Try it on a server of your own

The 402 setup above runs on any Linux server with Nginx, and on a VPS you control every header, log and rule yourself. RS Computers runs KVM VPS and VDS servers in Amsterdam, Dublin and Prishtina with NVMe storage, unmetered traffic and their own IPv4 and IPv6 addresses; availability by city is on the plans page. Before you open the server to the internet, add HTTPS with our reverse proxy and SSL guide. If you want help setting up bot pricing or verification on your server, message us on Telegram and we will quote the job.

← All articles