← Back to Blog

Serve Markdown to AI Agents from Nginx or Caddy: Self-Hosted Markdown for Agents

Published · by RS Computers

Nginx Caddy AI agents

To serve Markdown to AI agents from your own server, keep a Markdown copy of each page next to its HTML file and let the web server pick one by the request's Accept header. A browser asks for text/html and gets the normal page. An agent that sends Accept: text/markdown gets the clean .md file from the same URL. In Nginx that is one map on $http_accept plus try_files; in Caddy it is a header matcher plus rewrite. Add Vary: Accept so caches keep the two versions apart. This is the self-hosted version of Cloudflare's Markdown for Agents, and on a real 101 KB article from our own blog it cut the token count from 32,711 to 9,021 in our lab.

Key facts, checked on 13 October 2026:

What "Markdown for agents" means, in plain words

A web page is HTML: the text you read plus menus, footer links, scripts, CSS classes, icons and tracking tags. A language model reads all of it as tokens, the small chunks of text it counts and bills by. Markdown is plain text with a few symbols for structure, like # for a heading and - for a list item. It keeps the headings, lists, tables, links and code blocks and drops everything else.

Content negotiation is the old HTTP idea that one URL can have several versions. The client says what it prefers in the Accept header, and the server picks. Browsers send something like text/html,application/xhtml+xml,...,*/*;q=0.8. An agent that wants Markdown sends text/markdown. Same address, two answers, no separate "AI site" to maintain.

Our own site, rscomputers-ks.com, already does this through Cloudflare. Ask any article with curl -H 'Accept: text/markdown' and you get Content-Type: text/markdown, Vary: accept and a Content-Signal: ai-train=yes, search=yes, ai-input=yes header. Our homepage also sends a Link header that points agents to our llms.txt. This guide shows how to get the same result without a CDN, on a plain Debian server.

Before and after: the token table from our lab

We used two long articles from this blog as sample input: the reverse proxy guide and the MCP server guide. We counted tokens with OpenAI's tiktoken 0.14.0 and the o200k_base encoding. Other models split text a little differently, but the ratio holds.

Version of the reverse proxy guideBytesTokensSaving vs HTML
Raw HTML, as a browser gets it101,70232,711-
Pandoc on the whole page, no trimming46,82715,47453%
Cloudflare Markdown for Agents (live site)38,68510,36068%
Our script: only the <main> element, then pandoc35,3909,02172%

The MCP guide gave the same picture: 30,470 tokens as HTML, 8,624 through Cloudflare and 7,690 from our script, a 75% saving. Two things stood out. First, converting the whole page is not enough: pandoc kept our navigation, the footer and a base64 SVG icon sprite, which cost over 6,000 tokens on its own. Cutting to the <main> element first is what wins. Second, Cloudflare's version is a bit larger because it converts everything the page sends, including the menu, while our script throws the menu away before pandoc sees it.

Over the wire the gap is smaller, because gzip already squeezes HTML well. With Nginx's default gzip the HTML went out as 35,684 bytes and the Markdown as 15,287 bytes. Bandwidth is not the point here. Context window space and the agent's token bill are.

Step 1: generate the Markdown files

You need a .md file beside every .html page. If your site comes from a static site generator such as Hugo, Jekyll or Astro, the cleanest source is the Markdown you already write: copy it to the output folder at build time. For a site that only exists as HTML, convert it. We used pandoc, which Debian 13 ships as version 3.1.11.1.

We tested on a Debian 13 container limited to 1 GB of RAM and 2 CPU cores, on our Amsterdam lab server. Install the tools and create a normal user who owns the website files:

# as root
apt-get update
apt-get install -y nginx pandoc curl python3
adduser --disabled-password --gecos "" web
mkdir -p /var/www/site/guide
chown -R web:web /var/www/site
nginx -v
pandoc --version | head -1

Expected output: nginx version: nginx/1.26.3 and pandoc 3.1.11.1. The install took 7.6 seconds in our lab, and the container used 22 MB of RAM with Nginx running.

Now save this script as /usr/local/bin/make-md.sh, starting the file at the #!/bin/sh line (the line above it is our label), and make it executable with chmod 755 /usr/local/bin/make-md.sh. It walks a folder, keeps only what sits inside <main> and lets pandoc turn that into GitHub-style Markdown with no leftover HTML tags. If your theme wraps articles in <article> instead, change the word in the regular expression.

# as root: file /usr/local/bin/make-md.sh
#!/bin/sh
# make-md.sh: write page.md next to every page.html under a folder
# usage: make-md.sh /var/www/site
set -e
find "$1" -name '*.html' | while read -r page; do
  out="${page%.html}.md"
  # keep only what is inside <main>...</main> (menus, footer and inline SVG go away)
  python3 -c 'import re,sys
s=open(sys.argv[1],encoding="utf-8").read()
m=re.search(r"<main[^>]*>(.*?)</main>",s,re.S)
sys.stdout.write(m.group(1) if m else s)' "$page" \
  | pandoc -f html -t gfm-raw_html --wrap=none -o "$out"
  echo "$out"
done

Put a page in place and run the script as the web user. We used one of our own articles as the sample:

# as the web user (su - web)
curl -s -o /var/www/site/guide/index.html https://rscomputers-ks.com/reverse-proxy-ssl-caddy-nginx-traefik/
make-md.sh /var/www/site
ls -l /var/www/site/guide

Expected output: the script prints /var/www/site/guide/index.md, and ls shows index.html at 101,702 bytes and index.md at 35,390 bytes. Two pages took 0.44 seconds, so even a site with thousands of pages converts in a minute or two. Run the script again after every deploy, or add it to your build step, so the Markdown never goes stale.

Step 2: content negotiation in Nginx

Debian's Nginx does not know the .md extension; grep markdown /etc/nginx/mime.types returns nothing. The config below teaches it, picks the Markdown file when asked, and adds the headers. Because files in sites-enabled are read inside Nginx's http block, the map and types blocks can live in the same file.

# as root: file /etc/nginx/sites-available/markdown-site
# Pick the Markdown file when the client asks for text/markdown
# (but not when it says q=0, which means "never send me this").
map $http_accept $md_index {
    default                                 "";
    "~*text/markdown\s*;\s*q=0(\.0*)?(?![.0-9])"  "";
    "~*text/markdown"                       "index.md";
}

# For /guide/index.html, advertise /guide/index.md in a Link header.
map $uri $md_alternate {
    default                          "";
    "~^(?<dir>.*/)index\.html$"      "<${dir}index.md>; rel=\"alternate\"; type=\"text/markdown\"";
}

# Adds .md to the MIME types nginx already knows (this file sits in the http block).
types {
    text/markdown md;
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    root /var/www/site;
    index index.html;

    charset utf-8;
    charset_types text/markdown;
    gzip_types text/markdown;

    location / {
        add_header Vary Accept always;
        add_header Link "</llms.txt>; rel=\"describedby\"; type=\"text/markdown\"" always;
        add_header Link $md_alternate always;
        try_files $uri$md_index $uri $uri/ =404;
    }
}

How it works: for a normal request $md_index is empty, so try_files behaves as usual and the index directive serves index.html. For a Markdown request on /guide/, the first try becomes /guide/index.md, which exists, so Nginx sends it. A request for /style.css with a Markdown header tries /style.cssindex.md, finds nothing and falls through to the real file. An empty add_header value is skipped, so the alternate link only appears on HTML pages.

Enable the site and reload:

# as root
rm /etc/nginx/sites-enabled/default
ln -s /etc/nginx/sites-available/markdown-site /etc/nginx/sites-enabled/
nginx -t
systemctl reload nginx

Expected output from nginx -t: syntax is ok and test is successful. Our first draft also listed text/html under charset_types, which made Nginx warn about a "duplicate MIME type". It is already in the default list, so leave it out.

Test it with curl

# as root
curl -sI http://localhost/guide/
curl -sI -H 'Accept: text/markdown' http://localhost/guide/

What we saw, trimmed to the lines that matter:

# output we saw (first request, a browser-style client)
Content-Type: text/html; charset=utf-8
Content-Length: 101702
Vary: Accept
Link: </llms.txt>; rel="describedby"; type="text/markdown"
Link: </guide/index.md>; rel="alternate"; type="text/markdown"

# output we saw (second request, an agent)
Content-Type: text/markdown; charset=utf-8
Content-Length: 35390
Vary: Accept
Link: </llms.txt>; rel="describedby"; type="text/markdown"

We also tried the awkward cases. */* and a full Chrome-style Accept string both got HTML. TEXT/MARKDOWN got Markdown, because the ~* regex ignores case. The direct URL /guide/index.md worked too, which helps agents that follow the alternate link instead of sending the header.

The q=0 trap

Our first map was a single line, "~*text/markdown" "index.md". It sent Markdown even to Accept: text/markdown;q=0, which in HTTP means the client refuses Markdown. A regex ending in (,|$) did not fix it inside the map either. The lookahead version above did: q=0 and q=0.0 now get HTML, while q=0.5 still gets Markdown. Real clients rarely send q=0, but the test costs nothing. This is still a pragmatic shortcut, not a full parser that ranks every q-value; a client that sends text/html;q=1, text/markdown;q=0.1 will get Markdown.

Step 3: the same thing in Caddy

Debian 13 carries Caddy 2.6.2, which is old. We used Caddy's official package repository, as the Caddy install docs describe, and stopped Nginx first so the two did not fight over port 80:

# as root
systemctl disable --now nginx
apt-get install -y debian-keyring debian-archive-keyring apt-transport-https gnupg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' > /etc/apt/sources.list.d/caddy-stable.list
chmod o+r /usr/share/keyrings/caddy-stable-archive-keyring.gpg /etc/apt/sources.list.d/caddy-stable.list
apt-get update
apt-get install -y caddy
caddy version

Expected output: v2.11.7 followed by a hash, the version we got on 13 October 2026. Then replace the Caddyfile:

# as root: file /etc/caddy/Caddyfile
:80 {
	root * /var/www/site
	encode gzip

	# Clients that ask for text/markdown (and not with q=0) get index.md, if it exists.
	@markdown {
		header_regexp Accept (?i)text/markdown
		not header_regexp Accept (?i)text/markdown\s*;\s*q=0(\.0*)?([^.0-9]|$)
		file {path}index.md
	}
	rewrite @markdown {path}index.md

	@mdfile path *.md
	header @mdfile Content-Type "text/markdown; charset=utf-8"

	header Vary Accept
	header Link "</llms.txt>; rel=\"describedby\"; type=\"text/markdown\""
	file_server
}

The named matcher @markdown needs all three conditions: the header asks for Markdown, it does not say q=0, and the .md file really exists. Caddy uses Go's regular expressions, which have no lookahead, so the q=0 rule ends with ([^.0-9]|$) instead. In our test q=0 got HTML and q=0.5 got Markdown. Validate and reload:

# as root
caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy
curl -sI -H 'Accept: text/markdown' http://localhost/guide/

Expected output: Valid configuration, then Content-Type: text/markdown; charset=utf-8, Content-Length: 35390, Vary: Accept and Vary: Accept-Encoding. Caddy adds the second Vary itself because of encode gzip, and its gzip made the Markdown 13,886 bytes. The same q=0, */* and upper-case tests gave the same answers as Nginx. Caddy's process used 53 MB of RAM against about 21 MB for Nginx's master and two workers, a difference that only matters on the smallest servers. For HTTPS, swap :80 for your domain name and Caddy fetches a certificate on its own; our reverse proxy and free SSL guide covers that part for both servers.

Vary: Accept and caches

Vary: Accept tells every cache between you and the client that the answer depends on the Accept header. Without it, a cache can store the Markdown an agent got and hand it to the next browser, which then shows plain text. Both configs above send it on every response, HTML included, because a cache has to know about the variation before it stores the first copy.

There is a catch with CDNs. Cloudflare ignores most Vary values unless you configure them, so a negotiating origin behind a default Cloudflare setup can leak one version to the wrong visitor. If you already use Cloudflare on a Pro plan or above, turning on its own Markdown for Agents is simpler than negotiating at the origin. If you run without a CDN, as most small VPS sites do, Nginx and Caddy handle Vary correctly by themselves. Test with curl rather than a browser, because curl shows you the exact headers each version carries.

Negotiation only helps clients that ask. Two small additions help the rest:

The lab llms.txt was this short. The first line of the block below is our label; the file itself starts at # Lab site:

# as root: file /var/www/site/llms.txt
# Lab site

> Test site for Markdown content negotiation.

## Docs

- [Reverse proxy guide](/guide/index.md): Caddy, Nginx and Traefik with HTTPS

Neither llms.txt nor Markdown versions decide whether an AI company may train on your text. That is a policy question for robots.txt and the newer Content Signals, which our Content Signals, robots.txt and llms.txt guide covers. If you want agents to query your site rather than read it, see NLWeb with an MCP endpoint and our MCP server guide.

Measure your own savings

Character count divided by four is a common rough guess for tokens. On our Markdown it was close (8,845 against 9,021 real tokens), but on HTML it was 22% too low, because markup splits into many short tokens. To get real numbers, install tiktoken in a Python virtual environment and run a tiny counter:

# as root
apt-get install -y python3-venv

# as the web user
python3 -m venv ~/tok
~/tok/bin/pip install tiktoken
# file ~/count-tokens.py, written and run by the web user
# count-tokens.py: bytes, words and tokens (OpenAI o200k_base) per file
import sys, tiktoken
enc = tiktoken.get_encoding("o200k_base")
for path in sys.argv[1:]:
    text = open(path, encoding="utf-8").read()
    print(f"{path:40} {len(text.encode()):>7} bytes {len(text.split()):>6} words "
          f"{len(enc.encode(text)):>7} tokens  ~{len(text)//4:>6} (chars/4)")
# as the web user
~/tok/bin/python count-tokens.py /var/www/site/guide/index.html /var/www/site/guide/index.md

Expected output for our sample: 32,711 tokens for the HTML and 9,021 for the Markdown. tiktoken downloads its encoding file the first time it runs, so the server needs outbound HTTPS. Try it on your longest page; documentation and blog posts with heavy themes usually save the most.

Frequently asked questions

Do I need Cloudflare to serve Markdown to AI agents?

No. Any web server that can choose a file by request header can do it. Nginx needs a map on $http_accept and try_files; Caddy needs a header_regexp matcher and rewrite. You generate the .md files yourself with pandoc, a small script or your static site generator.

How many tokens does Markdown save compared to HTML?

Cloudflare measured about 80% on its own blog post, 16,180 tokens down to 3,150. In our lab two long articles dropped by 72% and 75% when we converted only the main content. Converting the whole page, menus included, saved just 53%.

Which AI agents send Accept: text/markdown?

Cloudflare names Claude Code and OpenCode as coding agents that send it. Many crawlers still ask for HTML, which is why a rel="alternate" link and an llms.txt that points at .md files are worth adding alongside negotiation.

Why do I need the Vary: Accept header?

It tells caches that one URL has more than one version depending on the Accept header. Without it a shared cache can store the Markdown copy and serve it to browsers, or the reverse. Send it on both the HTML and the Markdown responses.

Can I just publish .md URLs instead of using the Accept header?

Yes, and it is a good fallback. The llms.txt spec suggests a Markdown copy at the same address with .md added, and in our Nginx setup /guide/index.md answered directly with status 200. Agents that never send Accept: text/markdown can still find it through llms.txt or a rel="alternate" link.

What is the difference between llms.txt and a Markdown version of a page?

llms.txt is one file at the site root that summarises the site and links to its important pages. A Markdown version is a clean copy of a single page. llms.txt is the map and the .md files are the destinations, so the best setup has both.

Try it on a server you control

Everything here ran in a Debian 13 container capped at 1 GB of RAM, and Nginx with the whole site used about 22 MB of it, so a VPS Nano is enough for a static site and a VPS Mini leaves room for an app behind the same proxy. RS Computers runs KVM VPS and VDS servers in Amsterdam, Dublin and Prishtina, each with NVMe storage, unmetered traffic and its own IPv4 and IPv6 address; availability by city is on the plans page. See the VPS and VDS plans, harden the box with our Debian 13 security checklist, and if you would like us to set up the Nginx or Caddy side for you, ask for a quote on Telegram.

Related articles

← All articles

Chat on Telegram