Self-Hosted AI Gateway on a VPS: OmniRoute vs freellmapi vs LiteLLM, Tested on CPU

A self-hosted AI gateway is a small proxy on your own server that gives all your apps one OpenAI-compatible URL and one key, and then sends each request to whichever model provider you configured, switching to another one when the first fails. The three open-source options people compare in October 2026 are LiteLLM, OmniRoute and freellmapi. None of them needs a graphics card, because they route requests and never run the model themselves. We installed all three on a 2-core, 2 GB Debian 13 test server with a tiny local model behind them, broke the main upstream on purpose, and measured what each one costs in RAM and latency. LiteLLM is the one for teams, OmniRoute is the one with the most routing features, and freellmapi is by far the lightest.

Key facts, checked on 15 October 2026:

• LiteLLM has 61.3k GitHub stars and its latest PyPI release is 1.105.1. The code is MIT-licensed, except an enterprise/ folder that has its own commercial licence (BerriAI/litellm, PyPI).
• OmniRoute has 76.8k stars, an MIT licence and version 3.8.51, released on 30 September 2026. Its npm package needs Node.js 22.22.2 or a later 22.x release, or Node 24 to 26 (npm registry, diegosouzapw/OmniRoute).
• freellmapi has 35.6k stars, an MIT licence and version 0.14.2, released on 13 October 2026. Its README says it is "for personal experimentation and learning, not production" and single-user by design (tashfeenahmed/freellmapi).
• LiteLLM's production guide asks for 1 vCPU and 4 GiB of memory per worker and calls 4 GiB "a floor rather than a target" (LiteLLM docs). In our much lighter lab test, one worker used about 680 to 740 MB.
• In our lab, freellmapi sat at 92 MB of RAM when idle, OmniRoute at 560 MB and LiteLLM with its database at about 740 MB.

What a gateway does, and why it lives on a server

Without a gateway, every app holds its own API keys and its own idea of which model to call. An n8n workflow has one key, a Telegram bot has another, and a coding agent has a third. When a provider goes down or you hit a rate limit, each app fails in its own way. A gateway sits in the middle and handles four jobs:

• One endpoint. Apps talk to http://your-server:4000/v1 in the OpenAI format, whatever the model behind it.
• Fallback. If the first provider returns an error or times out, the gateway tries the next one in your list.
• Keys and limits. You give each app its own gateway key, with its own request limit and budget, and the real provider keys never leave the server.
• Logs. One place shows who called what, how many tokens it used and what it cost.

A VPS with a fixed IPv4 address suits this better than a laptop. The gateway has to be online whenever your apps are, and a single fixed address is easy to allow-list in firewalls and in your own services. Your keys sit on one machine you control instead of in five config files. It can share the box with other AI tools, such as an MCP server for Claude or ChatGPT.

A word on free tiers. OmniRoute and freellmapi are popular partly because they can chain many providers' free plans. Each provider sets its own terms for free usage, and those terms apply to you whatever the router does, so read them before you add a key. In this test we used no external provider at all.

LiteLLM vs OmniRoute vs freellmapi at a glance

The request rates come from ApacheBench against an instant fake upstream, so they measure the gateway alone on 2 vCPUs. A real model is far slower than any of these numbers. A 0.5B model on two CPU cores answers a few short requests per second at most.

Our lab setup

We used a Debian 13 container with 2 vCPUs and 2 GB of RAM on our Amsterdam test host, an AMD EPYC 7742 whose 8 cores were shared with other test labs that day. That is the same size as a VPS Micro (2 vCPU, 2 GB). Everything ran as a normal user called ai, bound to 127.0.0.1, with no Docker. Behind each gateway we placed:

• Ollama 0.40.4 with qwen2.5:0.5b, a 397 MB model, as the working upstream.
• An endpoint at 127.0.0.1:9, where nothing listens, as the "provider that is down".

The results come from our lab. Speeds on a real plan will differ, both because the host was busy and because a VDS has dedicated cores.

The CPU trap we hit first

Our first Ollama test produced 0.1 tokens per second. The cause was the container: Ollama saw all 8 host cores and started 8 threads on a 2-core limit. Pinning it to 2 threads gave 34 tokens per second. On a KVM server the guest only sees its own vCPUs, so this mostly matters inside Docker or LXC with a CPU limit. The fix is a model variant with a fixed thread count:

# as root
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:0.5b
printf "FROM qwen2.5:0.5b\nPARAMETER num_thread 2\n" > /root/Modelfile
ollama create qwen-cpu -f /root/Modelfile

With two threads, qwen2.5:0.5b ran at 31 to 34 tokens per second and qwen2.5:1.5b at 13 to 13.5. That is plenty for a fallback model that answers short questions when the cloud provider is down. For bigger local models on CPU, see our guide to running an LLM on a VPS without a GPU.

Tested setup 1: LiteLLM proxy with fallback and per-app keys

LiteLLM works without a database, but then it has no virtual keys, no per-key limits and no spend log. Without a database, a request with no key got HTTP 500 and a wrong key got HTTP 400, instead of a plain 401. Both were refused, but with a database the errors are clearer and you get the useful features, so we installed PostgreSQL 17.

# as root
apt-get install -y curl python3 python3-venv postgresql
adduser --disabled-password --gecos "" ai

# as the ai user
python3 -m venv ~/litellm
~/litellm/bin/pip install "litellm[proxy]==1.105.1"
~/litellm/bin/pip install prisma
~/litellm/bin/litellm --version

Expected output: LiteLLM: Current Version = 1.105.1. The install took 46 seconds in our lab. Now create the database and the secrets file. The master key is the admin password for the API.

# as root
echo "LITELLM_MASTER_KEY=sk-$(openssl rand -hex 16)" > /home/ai/litellm.env
PW=$(openssl rand -hex 12)
su - postgres -c "psql -c \"CREATE USER litellm WITH PASSWORD '$PW';\" -c \"CREATE DATABASE litellm OWNER litellm;\""
echo "DATABASE_URL=postgresql://litellm:$PW@127.0.0.1:5432/litellm" >> /home/ai/litellm.env
chown ai:ai /home/ai/litellm.env && chmod 600 /home/ai/litellm.env

A step the quick-start pages skip: without it, the proxy restarted in a loop with "Unable to find Prisma binaries". Generate the database client once:

# as the ai user
cd ~/litellm/lib/python3.13/site-packages/litellm/proxy
PATH=$HOME/litellm/bin:$PATH prisma generate

Expected output ends with Generated Prisma Client Python (v0.15.0). Next, the config. chat-main points at the dead endpoint and chat-local at Ollama. The fallback rule says to try chat-local whenever chat-main fails.

# as the ai user, file ~/litellm-config.yaml
model_list:
  - model_name: chat-main            # what clients ask for
    litellm_params:
      model: openai/some-model       # a provider that is down
      api_base: http://127.0.0.1:9/v1
      api_key: none
      timeout: 5
  - model_name: chat-local           # the fallback: Ollama on this server
    litellm_params:
      model: ollama_chat/qwen-cpu
      api_base: http://127.0.0.1:11434

router_settings:
  num_retries: 0
  fallbacks: [{"chat-main": ["chat-local"]}]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

# as root, file /etc/systemd/system/litellm.service
[Unit]
Description=LiteLLM proxy
After=network-online.target ollama.service

[Service]
User=ai
Environment=PATH=/home/ai/litellm/bin:/usr/local/bin:/usr/bin:/bin
EnvironmentFile=/home/ai/litellm.env
ExecStart=/home/ai/litellm/bin/litellm --config /home/ai/litellm-config.yaml --host 127.0.0.1 --port 4000
Restart=on-failure

[Install]
WantedBy=multi-user.target

# as root
systemctl daemon-reload
systemctl enable --now litellm
curl -s http://127.0.0.1:4000/health/readiness

The first start ran the database migrations and took 24 seconds. Then the check printed {"status":"healthy","db":"connected"}. Now ask for chat-main and watch it fall back:

# as root
set -a; . /home/ai/litellm.env; set +a
curl -s -D /tmp/h http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"model":"chat-main","messages":[{"role":"user","content":"Name the capital of Ireland in one word."}],"max_tokens":20}'
grep -i x-litellm /tmp/h

The answer was "Dublin." from ollama_chat/qwen-cpu, and the headers included x-litellm-model-group: chat-local and x-litellm-attempted-fallbacks: 1. Next, give an app its own key with a limit of 5 requests per minute:

# as root
curl -s http://127.0.0.1:4000/key/generate -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"key_alias":"n8n-bot","models":["chat-main","chat-local"],"rpm_limit":5,"max_budget":2}'

Seven quick requests with that key returned 200 five times and then 429, with the message "Rate limit exceeded ... Current limit: 5, Remaining: 0". A made-up key got 401. Requests and token counts are written to the LiteLLM_SpendLogs table, and the admin UI answers at /ui. Spend for the local model shows $0 because Ollama has no price.

What LiteLLM cost in our lab

• RAM: about 680 to 740 MB for the proxy with one worker, plus 51 MB for PostgreSQL. With two workers and Ollama loaded, the 2 GB container ran out of memory and the load test stalled.
• Added latency: a 1-token reply took 247 ms through LiteLLM and 217 ms straight from Ollama, so about 30 ms extra. The fallback hop to a refused port added about 17 ms more.
• Throughput: 61 requests per second at 20 parallel clients and 73 at 5, against the fake upstream.

One catch worth knowing: we also tried an upstream that never answers, at 10.255.255.1. Every request then waited the full 5-second timeout before falling back: in our test, LiteLLM tried the hanging upstream again on each request instead of skipping it. Keep timeout short on anything that has a fallback.

Tested setup 2: OmniRoute with a priority combo

OmniRoute calls a fallback chain a "combo" and offers 14 strategies on the command line, from simple priority to cost-optimised. We installed it with npm, which needs a newer Node.js than Debian 13 ships (20.19), so we took Node 24 from nodejs.org and checked its checksum:

# as root
V=v24.22.0
curl -fsSL https://nodejs.org/dist/$V/node-$V-linux-x64.tar.xz -o /tmp/node.tar.xz
curl -fsSL https://nodejs.org/dist/$V/SHASUMS256.txt | grep "linux-x64.tar.xz" | sed "s|node-$V-linux-x64.tar.xz|/tmp/node.tar.xz|" | sha256sum -c
tar -xJf /tmp/node.tar.xz -C /usr/local --strip-components=1
node -v

# as the ai user
npm config set prefix ~/.npm-global
npm install -g omniroute@3.8.51
mkdir -p ~/.omniroute && umask 077 && cat > ~/.omniroute/.env <<EOF
JWT_SECRET=$(openssl rand -hex 32)
API_KEY_SECRET=$(openssl rand -hex 32)
STORAGE_ENCRYPTION_KEY=$(openssl rand -hex 32)
INITIAL_PASSWORD=$(openssl rand -base64 18)
REQUIRE_API_KEY=true
OMNIROUTE_SERVER_HOST=127.0.0.1
PORT=20128
EOF

The npm install took 3 minutes 37 seconds and used 2.9 GB of disk. Two settings in that file matter. The packaged defaults include INITIAL_PASSWORD=CHANGEME and REQUIRE_API_KEY=false, so leaving them alone on a public server is a bad idea. With our values, a request without a key got 401 Authentication required. We started it with a systemd unit like LiteLLM's, using ExecStart=/home/ai/.npm-global/bin/omniroute serve --no-open --no-tray. It was ready in 8.5 seconds.

# as the ai user
O=~/.npm-global/bin/omniroute
$O providers add ollama-local --name local-ollama --no-credential --default-model qwen-cpu \
  --provider-specific-data '{"baseUrl":"http://127.0.0.1:11434"}'
$O providers add lm-studio --name dead-upstream --no-credential --default-model some-model \
  --provider-specific-data '{"baseUrl":"http://127.0.0.1:9/v1"}'
$O combo create chat-main --strategy priority --models lm-studio/some-model,ollama-local/qwen-cpu

Expected output: Added provider connection twice and Combo created: chat-main. Client keys are created through the API after logging in with the initial password:

# as root
PW=$(grep ^INITIAL_PASSWORD= /home/ai/.omniroute/.env | cut -d= -f2-)
curl -s -c /tmp/cj http://127.0.0.1:20128/api/auth/login -H "Content-Type: application/json" -d "{\"password\":\"$PW\"}"
curl -s -b /tmp/cj http://127.0.0.1:20128/api/keys -H "Content-Type: application/json" -d '{"name":"n8n-bot"}'

The login returned {"success":true} and the second call returned the new sk-... key with HTTP 201. Requests to the model chat-main came back from Ollama with the headers x-omniroute-fallback-attempts: 1 and x-omniroute-provider: ollama.

Two things we learned the hard way. First, connections of the same provider type are pooled. When our dead endpoint and our test endpoint were both of type vllm, the combo quietly sent "dead" traffic to the working one, so give each upstream in a fallback test its own type. Second, OmniRoute handles a hanging provider better than LiteLLM. The first request to an unreachable address took 71 seconds, then OmniRoute marked it down and the next three requests took about 0.33 seconds each.

Our numbers: about 560 MB of RAM when idle, 0.3% CPU when idle, and 47 to 50 requests per second against the fake upstream. Added latency was the least consistent of the three. It was 14 ms against the fake upstream on a quiet host, about 100 ms in front of Ollama, and close to 300 ms in some runs. The CLI also lists a response cache, which we did not measure.

Tested setup 3: freellmapi, the 92 MB option

freellmapi is the smallest of the three. We built it from the release tag with the same Node 24:

# as the ai user
git clone --depth 1 --branch v0.14.2 https://github.com/tashfeenahmed/freellmapi.git ~/freellmapi
cd ~/freellmapi && npm ci && npm run build
umask 077; printf "NODE_ENV=production\nENCRYPTION_KEY=%s\nHOST=127.0.0.1\nPORT=3001\n" $(openssl rand -hex 32) > ~/freellmapi.env

Set HOST=127.0.0.1 yourself. The default is ::, which means every interface. The systemd unit runs /usr/local/bin/node server/dist/index.js from /home/ai/freellmapi with that environment file, and the server was up in 2 seconds. The first account can only be created from the machine itself or with a one-time code printed in the log. We created it with a call from the server:

# as root
curl -s http://127.0.0.1:3001/api/auth/setup -H "Content-Type: application/json" \
  -d '{"email":"admin@example.com","password":"use-a-long-random-one"}'
curl -s http://127.0.0.1:3001/api/keys/custom -H "Authorization: Bearer <token from the line above>" \
  -H "Content-Type: application/json" \
  -d '{"baseUrl":"http://127.0.0.1:11434/v1","model":"qwen-cpu","label":"local-ollama"}'

Both calls answered HTTP 201. The fallback order can be set with PUT /api/fallback or on the dashboard, and clients use the one shared key from /api/settings/api-key. With the dead endpoint first in the chain, the first request got a 502 ("All 1 routed attempt(s) failed") and the next requests went to Ollama while the dead endpoint sat in cooldown. After a restart it was tried again, and one more request failed the same way. An upstream returning 503 behaved alike. The README says the router retries the next model on 429 and 5xx errors, but with custom endpoints that did not happen in our test, so set your apps to retry once.

The lab numbers were the best of the three. It used 92 MB idle and 208 MB after load, added about 4 ms in front of Ollama and 5 ms against the fake upstream, and handled 208 to 228 requests per second. Out of the box, it refused 130 of 200 rapid requests from one IP because of its default limit of 120 per minute. Set PROXY_RATE_LIMIT_RPM if one busy app sits behind it.

Which plan and which city

A gateway alone is light. The model behind it is what decides the plan. RS Computers VPS plans (Nano, Micro, Mini) have shared vCPUs, and VDS plans (Small, Medium, Large) have dedicated vCPUs.

For the city, put the gateway in the city where the apps that call it run, since every request passes through it. In our own ping.pe test on 6 October 2026, the Amsterdam server answered London in 6.7 ms and Frankfurt in 6.6 ms. Prishtina answered Vienna in 18.6 ms and Frankfurt in 27.8 ms. WonderNetwork's published averages put Dublin about 11 ms from London. Availability by city is on the plans page.

If you also want a chat interface on top of the gateway, point Open WebUI at its /v1 URL. Our AI apps on a VPS overview shows what else fits on the same server. When the gateway has to be reachable from other servers, put HTTPS in front of it as shown in our Caddy, Nginx and Traefik guide, and keep the gateway itself on 127.0.0.1.

Frequently asked questions

Does an AI gateway need a GPU?

No. LiteLLM, OmniRoute and freellmapi only forward requests and keep logs, so they run on plain CPU cores. In our test on 2 shared vCPUs they used between 92 MB and about 740 MB of RAM. You only need a GPU if you want to run large local models fast behind the gateway.

Is LiteLLM free for commercial use?

The LiteLLM proxy code is MIT-licensed, which allows commercial use. Files in the repository's enterprise/ folder have a separate commercial licence, so check which folder a feature comes from before you rely on it in a business.

What is the difference between LiteLLM and OmniRoute?

LiteLLM is a Python proxy aimed at teams. It stores per-app keys, budgets and spend logs in PostgreSQL and has an admin UI. OmniRoute is a Node.js gateway with a much larger provider catalogue, 14 routing strategies, a response cache and tools for coding agents. In our lab, LiteLLM added about 30 ms per request and OmniRoute between 14 and 300 ms.

How much RAM does a self-hosted LLM gateway need?

In our lab, freellmapi used 92 MB when idle, OmniRoute about 560 MB, and LiteLLM about 740 MB plus 51 MB for PostgreSQL. A 1 GB VPS is enough for freellmapi and a 2 GB VPS is enough for one LiteLLM worker or OmniRoute. Add 0.5 to 1.2 GB if a small local model runs on the same server.

Can I use Ollama as a fallback in LiteLLM?

Yes. Add Ollama as its own model with model: ollama_chat/<name> and api_base: http://127.0.0.1:11434. Then list it under fallbacks for your main model. In our test, a request to a dead provider was answered by qwen2.5:0.5b on Ollama, with the header x-litellm-attempted-fallbacks: 1.

Is it allowed to route free API tiers through a gateway?

The gateway software is legal. What you may do with each provider's free tier depends on that provider's terms, and those terms differ from one provider to the next. Read them before you add a key, and stay within the limits each provider sets.

Start with one URL for all your models

Pick LiteLLM if several people or apps need their own keys and budgets, OmniRoute if you want the most routing options for coding agents, and freellmapi if you want something tiny for yourself. freellmapi fits a VPS Nano and LiteLLM or OmniRoute a VPS Micro (see all plans), while a VPS Mini or VDS Small gives you room for a local fallback model. If you'd like us to install and configure the gateway for you, we quote that per job. Ask on Telegram and tell us which providers and apps you want connected.

← All articles