Run an LLM on a VPS Without a GPU: Microsoft's BitNet 1-Bit Model Tested on CPU

Yes, you can run an LLM on a VPS without a GPU, and Microsoft's BitNet is one of the lightest ways to do it. We built bitnet.cpp on a Debian 13 server limited to 4 vCPUs and 6 GB of RAM, loaded the official BitNet b1.58 2B4T model and got about 20 tokens per second of generated text with 4 threads, using about 1.4 GB of RAM. That is faster than most people read. There is a catch that the many "100B model on a laptop" articles skip: the current build gives broken answers out of the box. A one-line fix and the right chat template solve it, and both are below, with every number measured in our own lab.

Key facts, checked on 15 October 2026:

• The microsoft/BitNet repository had about 41,200 stars and is MIT licensed. bitnet.cpp 1.0 came out on 17 October 2024, and the README's newest news item is dated 23 July 2026.
• BitNet b1.58 2B4T is the official model: about 2 billion parameters (llama.cpp counts 2.41 billion), trained on 4 trillion tokens, with a 4,096-token context and an MIT license (Hugging Face model card). The downloadable GGUF file is 1.19 GB.
• The model card lists 0.4 GB of non-embedding memory and 29 ms CPU decoding latency against 65 ms for Qwen2.5 1.5B, on a CPU it does not name. Its benchmark average is 54.19, against 55.23 for Qwen2.5 1.5B.
• The README asks for Python 3.10 or newer, CMake 3.22 or newer and Clang 18 or newer.
• Issue #588, opened on 19 July 2026 and still open on 13 October 2026, reports that the bundled llama.cpp code uses the wrong activation function for BitNet models, which wrecks answer quality.

What "1-bit" means, in plain words

A language model is mostly a huge table of numbers called weights. A normal model stores each weight in 16 bits, and the popular Q4_K_M format squeezes that to roughly 4 to 5 bits. In BitNet each weight can only be -1, 0 or +1, which needs about 1.58 bits (hence "b1.58"), and the model is trained like that from the start. The processor then mostly adds and subtracts instead of multiplying, which suits a CPU.

bitnet.cpp is Microsoft's runtime for these models: a modified copy of llama.cpp, the C++ engine behind Ollama, with special CPU kernels for the 1.58-bit format. The model card warns that the Python transformers library gives none of the speed gains, so use bitnet.cpp.

About the "100B model on one CPU" headline

The README says bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU at 5 to 7 tokens per second. That figure comes from a benchmark model with the shape of a 100B network, not from a model you can download and chat with. No 100B BitNet model is published. The largest models on the README's list are community conversions of Falcon3 at up to 10B parameters. If you came here hoping to run a GPT-class model on a cheap server, this is not that.

Our test setup

The lab was a Debian 13 container with limits.cpu=4 and limits.memory=6GiB, on a host with an AMD EPYC 7742 (8 cores, AVX2) that other test labs were using at the same time. The host's load average moved between 2 and 30 during our runs, so treat our numbers as a realistic middle, not a best case. Results come from our lab and your plan's speed can differ.

For comparison we ran Qwen2.5 1.5B Instruct in the common Q4_K_M format with upstream llama.cpp build b11645, the official prebuilt CPU binary from the llama.cpp releases page. Its file is 1.12 GB, almost the same size as the BitNet file, which makes it a fair rival. Ollama is built on the same engine. If you want to repeat the comparison, fetch both after Step 2, with the venv from Step 2 active:

# as the ai user
mkdir -p ~/llamacpp ~/models
cd ~/llamacpp
curl -sSL -o llama.tar.gz https://github.com/ggml-org/llama.cpp/releases/download/b11645/llama-b11645-bin-ubuntu-x64.tar.gz
tar xzf llama.tar.gz
cd ~/models
hf download Qwen/Qwen2.5-1.5B-Instruct-GGUF qwen2.5-1.5b-instruct-q4_k_m.gguf --local-dir .

The binaries take 45 MB and run on Debian 13 as they are.

Step 1: install the build tools

Everything runs on the CPU. bitnet.cpp also has a GPU kernel since May 2025, but our servers have no graphics cards, so this guide does not touch it.

# as root
apt-get update
apt-get install -y git cmake clang python3 python3-venv python3-pip curl jq unzip
useradd -m -s /bin/bash ai
clang --version | head -1
cmake --version | head -1

Expected output on Debian 13: Debian clang version 19.1.7 and cmake version 3.31.6, both above the README's minimums. The ai user runs the model, so it never needs root.

Step 2: build bitnet.cpp and download the model

The README suggests conda with Python 3.10. A plain Python virtual environment on Debian's Python 3.13.5 worked for us, and it is one less thing to install.

# as the ai user (su - ai)
git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet
python3 -m venv ~/venv
. ~/venv/bin/activate
pip install -r requirements.txt

The pip step took 3 minutes 29 seconds and pulled in PyTorch 2.11.0 for CPU. The environment uses 1.1 GB of disk. Then download the ready-made GGUF and build:

# as the ai user, inside ~/BitNet with the venv active
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

The last lines should be INFO:root:Compiling the code using CMake. and GGUF model already exists at models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf. Be patient: the script compiles on a single core, and it took 16 minutes 37 seconds in our lab. The folder then holds 1.8 GB, model included.

Step 3: the one-line fix the README does not mention

Our first chat test failed badly. Asked to explain a reverse proxy in two sentences, the model repeated "It is used to protect clients from clients" until it hit the token limit. The reason is issue #588: in July 2026 the bundled llama.cpp code was reorganised, and the BitNet model graph now uses the SiLU activation where BitNet b1.58 was trained with squared ReLU. Nothing crashes, so it is easy to miss. The fix proposed in the issue is one changed word:

# as the ai user, inside ~/BitNet
sed -i 's/LLM_FFN_SILU, LLM_FFN_PAR, il);/LLM_FFN_RELU_SQR, LLM_FFN_PAR, il);/' 3rdparty/llama.cpp/src/models/bitnet.cpp
grep -n "LLM_FFN_RELU_SQR, LLM_FFN_PAR" 3rdparty/llama.cpp/src/models/bitnet.cpp
cmake --build build --config Release -j4

The grep line should print 133: LLM_FFN_RELU_SQR, LLM_FFN_PAR, il);. The rebuild only touches a few files and took 5 seconds. If a later version of the repository fixes this upstream, the sed command simply changes nothing. One caution from the issue thread: a maintainer pointed out that older community BitNet conversions derived from 1bitLLM were trained with SiLU, so keep this patched build for the official 2B4T model and test other models separately.

The model file has a second, smaller problem: it lacks the "pre-tokenizer" setting, and the server warns GENERATION QUALITY WILL BE DEGRADED! on start. The option --override-kv tokenizer.ggml.pre=str:llama-bpe supplies it at load time. To check the effect of both fixes we measured perplexity, a standard score for how well a model predicts real text (lower is better), on the first 20 chunks of the WikiText-2 test set:

# as the ai user
mkdir -p ~/data && cd ~/data
curl -sSL -o wt.zip https://huggingface.co/datasets/ggml-org/ci/resolve/main/wikitext-2-raw-v1.zip
unzip -q -o wt.zip
cd ~/BitNet
./build/bin/llama-perplexity -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -f ~/data/wikitext-2-raw/wiki.test.raw -t 4 --chunks 20 --override-kv tokenizer.ggml.pre=str:llama-bpe

Each run took about 2 minutes 25 seconds on 4 threads. Our stock and fixed figures match those posted in the issue (118.73 and 19.60 on 20 chunks), so this is not something about our machine.

Step 4: the official benchmark, then a fair comparison

The README's own benchmark script wraps llama-bench:

# as the ai user, inside ~/BitNet with the venv active
python utils/e2e_benchmark.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -n 200 -p 256 -t 4

It printed pp256 19.04 ± 0.30 and tg200 19.35 ± 0.25 tokens per second. "pp" is prompt processing, the speed at which the model reads your question, and "tg" is text generation, the speed at which it writes the answer. The script forces a batch size of 1, so the prompt is read one token at a time and both numbers come out nearly equal. Real chat servers read prompts in batches, so for the comparison we called llama-bench directly with its default batch, three repetitions, and alternated the two models so both saw the same host load:

# as the ai user, inside ~/BitNet
for t in 1 2 4; do ./build/bin/llama-bench -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p 128 -n 64 -t $t -r 3; done
~/llamacpp/llama-b11645/llama-bench -m ~/models/qwen2.5-1.5b-instruct-q4_k_m.gguf -p 128 -n 64 -t 4 -r 3

The ranges cover two or three separate runs. The honest reading: on this AMD EPYC, BitNet was not faster than a normal 4-bit model of the same file size. Qwen2.5 wrote about 35% faster and both read prompts at the same speed. BitNet has 2.41 billion parameters against Qwen's 1.78 billion, and Microsoft's speedup figures (2.37x to 6.17x on x86) compare it with full-precision models, not with Q4_K_M. Each doubling of threads gave both models about 1.8 times the generation speed.

Answer quality: three identical prompts

We started an OpenAI-compatible server for each model on 4 threads with temperature 0, so the answers are repeatable. BitNet needs one more fix here: the chat template stored in the GGUF file ("Human:" and "BITNETAssistant:" followed by an end-of-text token) makes the model ignore your question and invent its own conversation. The README's run_inference.py -cnv chat uses that template, and in our test it answered questions nobody asked even after the activation fix. The template from the model's tokenizer_config.json works:

# as the ai user
cat > ~/bitnet-chat.jinja <<'EOF'
{{ bos_token }}{% for message in messages %}{{ message['role'] | capitalize + ': ' + message['content'] | trim + '<|eot_id|>' }}{% endfor %}{% if add_generation_prompt %}{{ 'Assistant: ' }}{% endif %}
EOF
cd ~/BitNet
./build/bin/llama-server -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf --override-kv tokenizer.ggml.pre=str:llama-bpe -t 4 -c 2048 --temp 0 --jinja --chat-template-file ~/bitnet-chat.jinja --host 127.0.0.1 --port 8080

Wait for llama_server: listening on http://127.0.0.1:8080, then ask from a second SSH session:

# as the ai user, second terminal
jq -n --arg p "Explain in two sentences what a reverse proxy is." '{messages:[{role:"user",content:$p}],max_tokens:256,temperature:0}' | curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @- | jq -r '.choices[0].message.content'

Three prompts prove little, but they fit the published scores: both models handle short, clear tasks and neither replaces a large cloud model. The model card also says support for languages other than English is limited.

RAM, disk and several users at once

The BitNet server used 1.39 GB of resident memory after loading with a 2,048-token context and 1.44 GB after our three prompts. The Qwen server used 1.83 to 1.85 GB with the same settings. That 0.4 GB difference is BitNet's real advantage on a small server. On disk, plan for about 4.1 GB: 1.8 GB for the BitNet folder with the model, 1.1 GB for the Python environment and room for logs and updates.

Then we sent identical requests in parallel to one BitNet server started with -np 4 (four slots) and a 4,096-token context, each asking for 128 tokens:

Four people together got about 42 tokens per second, more than twice what one person gets, and memory rose only to 1.58 GB. A small team sharing one model is realistic. A public chatbot with dozens of visitors at once is not.

Why dedicated vCPUs matter for CPU inference

llama.cpp splits every token across all its threads and waits for the slowest one before starting the next token. If one vCPU is busy with someone else's work for a moment, all threads wait. We saw this by accident: when two benchmarks ran at the same time on our 4 vCPUs, the 2-thread run fell to 5.2 tokens per second with a spread of 1.5, against a steady 11.2 when it ran alone. On a VPS, the vCPUs are shared with other customers, so the same thing can happen whenever neighbours are busy. On a VDS each vCPU is reserved for you, so the speed you measure stays far more consistent. We did not compare a real VPS Mini and VDS Small side by side; this explanation rests on how llama.cpp works and on that one accidental run. If you want to check a server, our VPS benchmark commands show how to measure CPU steal, and the VPS, VDS and dedicated server comparison explains the difference in more depth.

Which plan for which job

Our lab used 4 threads, so the VPS Mini and VDS Small rows rest on measured numbers. The 8 and 16 vCPU rows are estimates: generation speed on a CPU is limited by memory bandwidth as well as cores, so do not count on another 1.8 times for each doubling.

For the city, latency barely matters for a chat that writes 20 tokens per second, so choose by where your users and data are. Our own ping.pe measurements from 6 October 2026, published on our location pages, give 6.6 ms from Frankfurt and 6.7 ms from London to Amsterdam, and 18.6 ms from Vienna to Prishtina, where Skopje is about 3 ms away over our own network. For Dublin, WonderNetwork's published averages put London about 11 ms away.

Availability by city is on the plans page, and you can move to a bigger plan later from the client area with a short reboot.

What a CPU server will not do

• Run a GPT-4-class model. Models above roughly 8B parameters get slow on CPU, and the 100B BitNet figure is a benchmark shape, not a model.
• Serve many users at once. Our four parallel users each dropped to about half the single-user speed.
• Read long documents quickly. At about 90 tokens per second of prompt reading, a 3,000-token document takes over 30 seconds before the first word of the answer, and the model's context ends at 4,096 tokens.
• Train or fine-tune models. That needs GPUs, which our servers do not have.

For strong reasoning or long documents a hosted model API serves you better, and you can still run the interface on your own server: see our guide to AI apps on a VPS and the Open WebUI 0.12 on CPU test. The BitNet server above speaks the OpenAI-style API and listens only on 127.0.0.1; put a reverse proxy with HTTPS and a password in front before anyone else uses it.

Frequently asked questions

Can you run an LLM on a VPS without a GPU?

Yes. Small models with 1 to 3 billion parameters run well on a CPU. In our test on 4 vCPUs, Microsoft's BitNet b1.58 2B4T wrote about 20 tokens per second and Qwen2.5 1.5B in Q4_K_M format about 27, both using under 2 GB of RAM.

How much RAM does BitNet b1.58 2B4T need?

In our lab the bitnet.cpp server used 1.39 to 1.44 GB of RAM with a 2,048-token context, and 1.58 GB with a 4,096-token context and four users. The model file is 1.19 GB. A server with 4 GB of RAM runs it with room for the operating system.

Is BitNet faster than llama.cpp with a 4-bit model?

Not in our test. On an AMD EPYC 7742 with 4 threads, BitNet 2B4T generated 19.8 to 20.6 tokens per second and Qwen2.5 1.5B Q4_K_M 26.9 to 27.4, with equal prompt speed. BitNet used about 0.4 GB less RAM. Microsoft's speedup claims compare against full-precision models.

Why does BitNet give repeated or nonsense answers?

Since July 2026 the llama.cpp code bundled with bitnet.cpp uses the SiLU activation instead of squared ReLU for BitNet models (GitHub issue #588). Changing LLM_FFN_SILU to LLM_FFN_RELU_SQR in 3rdparty/llama.cpp/src/models/bitnet.cpp and rebuilding brought perplexity from 118.7 to 19.6 in our test. The chat template stored in the GGUF file also needs replacing with the one from the model's tokenizer_config.json.

Can I really run a 100B model on one CPU with bitnet.cpp?

No published 100B BitNet model exists. The README's figure of 5 to 7 tokens per second for 100B comes from a benchmark model with that shape, used to measure speed. The largest downloadable models on its list have up to 10 billion parameters.

Do I need a dedicated vCPU for a local LLM?

It helps. llama.cpp waits for its slowest thread on every token, so time lost to other customers on shared vCPUs slows the whole answer. A VDS with dedicated vCPUs gives steadier speed; a VPS with shared vCPUs is fine for testing and light personal use.

Try it on your own server

BitNet works on a CPU server today once you apply the activation fix and the correct chat template: about 20 tokens per second in 1.4 GB of RAM on 4 vCPUs, plenty for a private assistant, a Telegram bot or scripts that sort and summarise text. A normal 4-bit model was still faster on our hardware, so test both. Start with a VDS Small if the answers must arrive on time, or a VPS Mini to experiment, from the plans page. Questions about sizing, or want us to quote the setup for you? Message us on Telegram.

← All articles