Content Signals, robots.txt and llms.txt in 2026: How to Control What AI Does With Your Site

In 2026 you control what AI does with your site through three files and headers that do different jobs. robots.txt says which bots may fetch which URLs. A Content-Signal line inside robots.txt says what the content may be used for once it is fetched: search, live AI answers (ai-input) or model training (ai-train). llms.txt is a Markdown map that helps AI assistants find your best pages. None of them is a lock. They are public requests that well-behaved crawlers follow, so you pick a goal, write the right robots.txt for it, and use server rules for the bots that ignore it.

Key facts, checked on 13 October 2026:

• robots.txt is an IETF standard since September 2022 (RFC 9309), and the RFC itself says its rules "are not a form of access authorization".
• Only 4% of the 200,000 most-visited domains declared AI usage preferences with Content Signals, while 78% had a robots.txt (Cloudflare, 17 April 2026).
• Fewer than 1% of Cloudflare sites block search crawlers, but 17% use some mechanism to block AI training (Cloudflare, 15 September 2026).
• Training made up 52% of AI crawler requests by June 2026, up from 22% in spring 2025, measured by the crawlers' stated purpose (Cloudflare, 30 September 2026).
• Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google crawler docs, updated 14 July 2026).
• OpenAI and Perplexity both say robots.txt changes can take about 24 hours to show up in their systems (OpenAI, Perplexity).

Three layers, three different jobs

Most confusion comes from mixing these up, so here they are side by side.

A bot that respects robots.txt never reaches a blocked page, so the Content-Signal only matters for pages you allow. That is the whole point of it: you can let a crawler in for search and still say "not for training".

Content Signals: search, ai-input, ai-train

Content Signals is a policy published by Cloudflare at contentsignals.org. It adds one line to a robots.txt group with three yes/no values:

• search means building a search index and showing links and short excerpts. Cloudflare's definition says search "does not include providing AI-generated search summaries".
• ai-input means feeding your page into an AI model at answer time, for example retrieval or grounding for a generative search answer.
• ai-train means training or fine-tuning AI models.

The site is blunt about the limits: these preferences "do not, however, prevent crawler operators from taking your content at a technical level". Treat the line as a clearly stated licence wish, not as enforcement. If you leave a value out, you have expressed no preference for that use.

A standards track is running in parallel. The IETF AI Preferences working group is drafting a vocabulary and a robots.txt rule named Content-Usage (draft-ietf-aipref-vocab, revision 08 in September 2026). It is still a draft, so in October 2026 the Content-Signal line is the format you will actually see in the wild.

Our own robots.txt as a real example

This is what rscomputers-ks.com/robots.txt serves today. We want to be found and quoted, so all three uses are allowed:

# any user, any computer: fetch our live file
curl -s https://rscomputers-ks.com/robots.txt

# Content Signals (https://contentsignals.org): search results, live AI answers and AI training are all allowed
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /includes/
# WHMCS billing system (served at client.rscomputers-ks.com), not website content
Disallow: /client.rscomputers-ks.com/
# CRM (served at crm.rscomputers-ks.com), private
Disallow: /crm.rscomputers-ks.com/

Sitemap: https://rscomputers-ks.com/sitemap.xml

Every page also sends a Link header that points agents to the llms.txt file and the sitemap:

# any user: show the Link header on our homepage
curl -sI https://rscomputers-ks.com/ | grep -i ^link

link: </llms.txt>; rel="describedby"; type="text/markdown", </sitemap.xml>; rel="sitemap"; type="application/xml"

Our lab found a rule-order quirk in this very file. The parser section below explains it.

The robots.txt tokens that matter in 2026

A "token" is the name you write after User-agent:. The big operators now split their bots by job, which is what makes a "search yes, training no" policy possible. Everything below comes from each operator's own page, checked on 13 October 2026.

The pattern to remember: training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent) can be blocked with almost no search cost. Search tokens (OAI-SearchBot, Claude-SearchBot, PerplexityBot) decide whether AI assistants can cite you. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) act for a person and may not obey robots.txt at all.

Three ready robots.txt templates

Replace /admin/ with your private paths and example.com with your domain. We put Disallow lines above Allow: / on purpose; the parser section explains why.

Goal 1: maximum AI visibility

For a business that wants to be found, quoted and recommended by every assistant. This is the policy we use ourselves.

# Goal 1: maximum AI visibility
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Disallow: /admin/
Allow: /

Sitemap: https://example.com/sitemap.xml

Goal 2: search and AI answers, no training

The middle road for most publishers. Search engines and AI search bots can still find and cite you; the training-only tokens are shut out. If you do not want your pages used in live AI answers either, change ai-input=yes to ai-input=no.

# Goal 2: search and AI answers yes, model training no
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /admin/
Allow: /

# Crawlers and tokens that exist only for model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

Sitemap: https://example.com/sitemap.xml

Goal 3: classic search only, no AI crawlers

Google, Bing and Apple search keep working. Every AI-specific bot is told to stay out.

# Goal 3: classic search only, no AI crawlers at all
User-agent: *
Content-Signal: search=yes, ai-input=no, ai-train=no
Disallow: /admin/
Allow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

Sitemap: https://example.com/sitemap.xml

Several User-agent lines stacked over one rule form a single group, which RFC 9309 allows and both parsers in our lab understood.

Which template should you pick?

The SEO trade-offs nobody puts in the template

• Google AI Overviews and AI Mode are built from the normal Googlebot crawl. Blocking Google-Extended does not take you out of them. Google's AI features page points to nosnippet, data-nosnippet, max-snippet and noindex instead, and those also shrink your classic snippets.
• Blocking OAI-SearchBot removes you from ChatGPT search answers, though OpenAI says you can still appear as a plain navigational link.
• Blocking Googlebot, Bingbot or Applebot to stop AI also wipes out your search traffic. Use the dedicated tokens instead.
• Google says you "don't need to create new machine readable files, AI text files, or markup" to appear in its AI features. An llms.txt file will not raise your Google ranking, and it will not hurt it.

Why one parser says yes and another says no

We tested all three templates in our lab on a Debian 13 container (Debian 13.7, Python 3.13.5) with two parsers: Python's built-in urllib.robotparser and Google's open-source robotstxt library, the C++ code Googlebot uses. We checked 14 bot names against a normal page and a private path. With Disallow above Allow: /, both parsers gave identical answers for all 42 bot and template combinations.

To repeat it, build Google's parser as a normal user (it took 17 seconds on 2 vCPUs in our lab), then ask it about one bot and one URL. Here the Goal 1 template is saved as ~/max-visibility.txt:

# as a normal user (needs git, cmake, g++ and make installed by root)
git clone https://github.com/google/robotstxt.git
cd robotstxt && mkdir c-build && cd c-build
cmake .. -DROBOTS_BUILD_TESTS=OFF && make -j2
./robots ~/max-visibility.txt GPTBot https://example.com/admin/

user-agent 'GPTBot' with URI 'https://example.com/admin/': DISALLOWED

Our results for Goal 2, from the Google parser (Python matched it line for line):

Googlebot          allow       BLOCK
Bingbot            allow       BLOCK
Google-Extended    BLOCK       BLOCK
Applebot           allow       BLOCK
Applebot-Extended  BLOCK       BLOCK
GPTBot             BLOCK       BLOCK
OAI-SearchBot      allow       BLOCK
ChatGPT-User       allow       BLOCK
ClaudeBot          BLOCK       BLOCK
Claude-SearchBot   allow       BLOCK
Claude-User        allow       BLOCK
PerplexityBot      allow       BLOCK
Perplexity-User    allow       BLOCK
CCBot              BLOCK       BLOCK

(First column: /blog/post. Second column: /admin/.)

Gotcha 1: rule order

RFC 9309 says the longest matching rule wins, whatever its position. Python's parser uses the first rule that matches. With Allow: / first, the two disagree:

# as a normal user
printf 'User-agent: *\nAllow: /\nDisallow: /admin/\n' > order.txt
python3 -c 'import urllib.robotparser as r; p=r.RobotFileParser(); p.parse(open("order.txt").read().splitlines()); print(p.can_fetch("GPTBot","https://example.com/admin/"))'
~/robotstxt/c-build/robots order.txt GPTBot https://example.com/admin/

True
user-agent 'GPTBot' with URI 'https://example.com/admin/': DISALLOWED

Our own robots.txt lists Allow: / first. Google's parser correctly blocks /includes/; Python's parser lets it through. Big crawlers follow the RFC, but small scrapers built on simple libraries may not, so writing Disallow lines first costs nothing and removes the doubt.

Gotcha 2: a bot's own group replaces the * group

When a bot finds a group with its name, it ignores the * group completely. Rules are not inherited:

# as a normal user
printf 'User-agent: *\nDisallow: /admin/\n\nUser-agent: GPTBot\nAllow: /\n' > inherit.txt
~/robotstxt/c-build/robots inherit.txt GPTBot https://example.com/admin/
~/robotstxt/c-build/robots inherit.txt ClaudeBot https://example.com/admin/

user-agent 'GPTBot' with URI 'https://example.com/admin/': ALLOWED
user-agent 'ClaudeBot' with URI 'https://example.com/admin/': DISALLOWED

If you give a bot its own group, repeat every private path inside it. The same goes for the Content-Signal line: put it in each group where it should apply.

Serve llms.txt from Nginx with the right headers

The llms.txt proposal by Jeremy Howard (first published 3 September 2024) asks for a Markdown file at /llms.txt with an H1 title (the only required part), a one-paragraph summary in a blockquote, and H2 sections that list links. A section called "Optional" holds links an agent may skip. A small example:

# Example Hosting

> Example Hosting sells KVM VPS in three European cities. This file lists the pages an AI assistant should read first.

## Plans
- [All VPS plans](https://example.com/vps/): CPU, RAM, NVMe and traffic for each plan

## Guides
- [Secure a new server](https://example.com/blog/secure-server/): first 30 minutes on a fresh Debian 13 VPS

## Optional
- [Company history](https://example.com/about/)

Stock Nginx maps every .txt file to text/plain. That works, but text/markdown tells a client exactly what it is getting. This server block on Nginx 1.26.3 sets the type, adds UTF-8 and sends the Link header on every response:

# as root: /etc/nginx/sites-available/example
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name example.test;
    root /var/www/example;

    # Point agents to llms.txt on every response (RFC 8288 Link header)
    add_header Link '</llms.txt>; rel="describedby"; type="text/markdown"' always;

    location = /robots.txt {
        charset utf-8;
    }

    location = /llms.txt {
        types { }
        default_type text/markdown;
        charset_types text/markdown;
        charset utf-8;
    }
}

# as root: enable the site and remove Debian's default site, which also claims default_server
ln -sf /etc/nginx/sites-available/example /etc/nginx/sites-enabled/example
rm /etc/nginx/sites-enabled/default
nginx -t && systemctl reload nginx
curl -sI http://localhost/llms.txt

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Link: </llms.txt>; rel="describedby"; type="text/markdown"

Three things we hit in the lab. If you keep Debian's default site enabled, nginx -t fails with "a duplicate default server for 0.0.0.0:80", so either remove it as above or drop default_server from your own block. Without charset_types text/markdown; the header came back as plain text/markdown with no charset, because Nginx only adds a charset to a short default list of types. And a curl fired in the same second as systemctl reload got a 404; one second later it was a 200. The whole Nginx setup used about 20 MB of RAM (master 9 MB, two workers about 5.5 MB each), so this fits on the smallest plan. For HTTPS in front of it, follow our reverse proxy and SSL guide. If you want to go further and answer Accept: text/markdown for every page, see Markdown for AI agents with Nginx and Caddy.

What Cloudflare changed in 2026

If your site sits behind Cloudflare, part of this policy may already be written for you, so check the dashboard before you edit robots.txt by hand.

• AI Crawl Control (formerly AI Audit) shows which AI services fetch your pages, lets you allow or block each one, and tracks crawlers that break your robots.txt. It is "available on all plans" (Cloudflare docs, updated 14 August 2026).
• Search, Training and Agent categories. On 1 July 2026 Cloudflare split AI bots into these three groups for all customers, Free plan included, and announced that from 15 September 2026 new domains would block Training and Agent on pages that show ads, with Search left allowed. Mixed-use crawlers such as Googlebot, Applebot and Bingbot would follow the strictest rule that applies to them.
• What actually shipped on 15 September. Cloudflare refined the plan: new domains are offered one of two presets, a stricter one for ad-supported sites ("Block on pages with ads" for Agent and "Disallow AI Training" for Training) and an allow-all one for sites without ads. The new Disallow AI Training setting keeps "accountable" mixed-use crawlers allowed for search. The same day, "Block AI Bots" was deprecated and Managed robots.txt was replaced by Bot Preference Sync, which publishes the no-training preference in your robots.txt.
• Managed robots.txt had written a Content-Signal of search=yes,ai-train=no for you. If you use Cloudflare and also edit robots.txt on your server, check the file that visitors actually receive with curl.
• Pay Per Crawl to Pay Per Use. Cloudflare's 1 July 2026 press release says it "is evolving Pay Per Crawl into Pay Per Use", so publishers get paid when content is used in an answer, not when it is fetched. Its 30 September post says Pay Per Use launches as a beta. Background in our HTTP 402 article.
• Markdown for Agents (docs) converts HTML to Markdown at the edge when a client sends Accept: text/markdown.
• Agent Readiness score. The free scanner at isitagentready.com (launched 17 April 2026) checks robots.txt, sitemap, Link headers, Markdown negotiation, Content Signals, AI bot rules and more. llms.txt is only checked if you customise the scan.

How to test your setup

1. Fetch the file the world sees. A CDN, plugin or host panel can rewrite it, so test the public URL:

   # any user
   curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://example.com/robots.txt
   curl -s https://example.com/robots.txt | grep -i content-signal

   You want 200 text/plain. RFC 9309 says a 5xx answer must be read as "disallow everything", and a 4xx as "no rules", so a broken robots.txt can hide your whole site.
2. Check the rules with a parser. Use Google's robotstxt binary as shown above, one bot and one URL at a time.
3. Use the Search Console robots.txt report. In Google Search Console it lists the robots.txt files Google found for your top 20 hosts, when each was last checked, and parse issues. After a fix, use "Request a recrawl" from the menu next to the file.
4. Run the Agent Readiness scan at isitagentready.com and read which checks fail.
5. Watch your logs. Rules only matter if bots obey them. Count hits per user agent and compare with your robots.txt, as in our guide to AI crawler log analysis. For bots that ignore the file, a proof-of-work gate like Anubis blocks them at the server.

Allow about 24 hours after a change. OpenAI, Perplexity and Meta all say their systems may take that long, and RFC 9309 lets crawlers cache the file for up to 24 hours.

Frequently asked questions

What is a Content-Signal in robots.txt?

It is a line such as Content-Signal: search=yes, ai-input=yes, ai-train=no inside a robots.txt group. It tells crawlers what they may use your content for after they fetch it: a search index, live AI answers, or AI model training. It was published by Cloudflare at contentsignals.org and is a stated preference, not a technical block.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot only collects training data. ChatGPT search uses OAI-SearchBot, and user-requested page visits use ChatGPT-User. To leave ChatGPT search answers you must disallow OAI-SearchBot, which OpenAI says takes about 24 hours to apply.

Does blocking Google-Extended hurt my Google ranking?

No. Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls whether Google may use your content for Gemini training and grounding. It does not keep your pages out of AI Overviews, which come from the normal Googlebot crawl.

Is llms.txt worth adding in 2026?

It is cheap and does no harm, but expect little. Google says you do not need AI text files to appear in its AI features. Some AI tools and coding agents read llms.txt, and Cloudflare's Agent Readiness scanner can check it. Add it if you have a clear set of key pages to point to, and serve it as Markdown.

Can robots.txt actually stop AI scrapers?

Only the ones that choose to obey it. RFC 9309 says robots.txt is not access authorization, and OpenAI and Perplexity say their user-triggered fetchers may not follow it. To stop bots that ignore the file, block them at the web server or use a challenge such as Anubis.

How long does a robots.txt change take to work?

Usually within a day. RFC 9309 lets crawlers cache robots.txt for up to 24 hours, and OpenAI, Perplexity and Meta each mention about 24 hours. In Google Search Console you can request a recrawl of the robots.txt file to speed up Google.

Put the policy on a server you control

Pick a goal, copy the matching template, put Disallow lines before Allow, and test with a real parser and curl before you trust it. Then check your logs, because the bots that ignore robots.txt are the ones you will want to block at the server. A KVM VPS gives you full control of Nginx headers, logs and rate limits. Our plans run in Amsterdam, Dublin and Prishtina with NVMe storage, unmetered traffic and your own IPv4 and IPv6; see the plans page for availability by city, or ask us on Telegram which size fits your site.

← All articles