Build a Small Ceph Cluster on Three VPS: A Test Lab, Step by Step

You can build a working Ceph cluster on three ordinary VPS servers in about an hour: install cephadm on the first node, bootstrap it, add the other two, give each one an OSD, and you have replicated block storage (RBD), a shared filesystem (CephFS) and an S3-compatible object store (RGW) to experiment with. We built exactly that with Ceph Tentacle 20.2.4 on three Ubuntu 24.04 machines, including the workaround a single-disk VPS needs and a fix for a kernel problem the official guides do not mention yet.

What this lab gives you, and what we measured:

• Three monitors in quorum, three OSDs, replication across all three hosts (every object stored three times).
• Block storage, a shared filesystem and S3 working from the same cluster.
• With one node switched off, reads and writes kept working; after it came back, all 211 placement groups were healthy again within 50 seconds, with no manual steps.
• An honest limit: this is a lab for learning and testing, not production storage. Production Ceph wants dedicated disks, more memory per OSD and at least a 10 Gbit/s network.

What Ceph is, in one paragraph

Ceph is open-source software that turns many ordinary servers into one storage system with no single point of failure. Data is split into objects, and an algorithm called CRUSH decides which disks hold each copy, so there is no central lookup table to lose. Monitors (MON) keep the cluster map, managers (MGR) run the orchestration and dashboard, and OSD daemons store the data, usually one per disk. On top sit three interfaces: RBD for virtual disks, CephFS for a shared POSIX filesystem and RGW for S3 and Swift object storage. CERN runs well over 100 petabytes on it, and many public clouds build their block and object storage the same way. Our article on Ceph block and object storage explains the architecture in depth; this one is hands-on.

What you need

Why Ubuntu 24.04 and not Debian 13: Ceph's own package repository for Tentacle has builds for Ubuntu 24.04 (noble) but none for Debian 13 yet, and both distributions' own archives ship older Ceph versions, Squid 19.2.3 on Ubuntu and Reef 18.2.7, already past end of life, on Debian. We download cephadm straight from Ceph instead. Release dates come from the official Ceph releases page.

In the commands below, replace 203.0.113.11, 203.0.113.12 and 203.0.113.13 with the public IPv4 addresses of your three servers, and call them ceph1, ceph2 and ceph3.

Step 1: prepare all three servers

On each server, set its name, let the three find each other by name, and open the firewall to the other two:

# as root on ceph1 (use ceph2 and ceph3 on the others)
hostnamectl set-hostname ceph1
cat >> /etc/hosts <<'EOF'
203.0.113.11 ceph1
203.0.113.12 ceph2
203.0.113.13 ceph3
EOF
ufw allow 22/tcp
ufw allow from 203.0.113.11
ufw allow from 203.0.113.12
ufw allow from 203.0.113.13
ufw enable

Ceph talks on ports 3300 and 6789 for monitors and 6800 to 7568 for the other daemons; allowing the three nodes' addresses covers all of them without exposing them to the internet. Then install what cephadm needs, on each server:

# as root on every node
apt update
apt install -y podman lvm2 chrony curl
systemctl enable --now chrony

Chrony matters more than it looks. Monitors complain about clock skew above 0.05 seconds, and on virtual machines clocks drift.

Step 2: an OSD on a VPS with only one disk

Here is the catch most tutorials skip. cephadm only accepts an empty disk for an OSD: no partitions, no filesystem, at least 5 GB. A typical VPS has one disk, and it holds the operating system. Our workaround, tested and rebooted: an LVM volume inside a file, attached at boot before Ceph starts. On each server:

# as root on every node
fallocate -l 30G /srv/ceph-osd.img
LOOP=$(losetup -f --show /srv/ceph-osd.img)
pvcreate "$LOOP"
vgcreate ceph-lab "$LOOP"
lvcreate -l 100%FREE -n osd ceph-lab
lvs ceph-lab

The last command should show a volume called osd of about 30 GB. Now make sure it comes back after a restart:

# as root on every node
cat > /etc/systemd/system/ceph-loop.service <<'EOF'
[Unit]
Description=Attach the file-backed Ceph OSD volume
DefaultDependencies=no
After=local-fs.target
Before=ceph.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'losetup -j /srv/ceph-osd.img | grep -q . || losetup -f /srv/ceph-osd.img'
ExecStart=/sbin/vgchange -ay ceph-lab

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable ceph-loop.service

Size the file with the monitor in mind: it warns "low on available space" when less than 30% of the system disk is free. On an 80 GB VPS, a 30 GB file leaves plenty; in our first attempt, an 8 GB file on a 16 GB disk triggered the warning. A file-backed OSD is slower than a real disk and is for learning only.

Step 3: install cephadm and bootstrap the first node

On ceph1 only:

# as root on ceph1
curl --silent --remote-name --location https://download.ceph.com/rpm-tentacle/el9/noarch/cephadm
chmod +x cephadm
./cephadm add-repo --release tentacle
./cephadm install
cephadm install ceph-common
cephadm bootstrap --mon-ip 203.0.113.11 --skip-monitoring-stack

The file says el9, but it is a Python program that runs on Ubuntu too. Bootstrap pulls the Ceph container image, starts the first monitor and manager, and ends with Bootstrap complete., a dashboard address on port 8443 and a one-time admin password. --skip-monitoring-stack leaves out Prometheus and Grafana to save memory on small servers. Check it:

# as root on ceph1
ceph --version
ceph -s

Expect ceph version 20.2.4 ... tentacle (stable), one monitor and one manager. The health line shows warnings until the OSDs exist; that is normal.

Step 4: add the other two servers and the OSDs

cephadm manages the other hosts over SSH as root. Copy the cluster's key to them, then add them:

# as root on ceph1
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph2
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph3
ceph orch host add ceph2 203.0.113.12
ceph orch host add ceph3 203.0.113.13
ceph orch host ls

If root login with a password is switched off on ceph2 or ceph3, ssh-copy-id fails; in that case paste the one line from /etc/ceph/ceph.pub into /root/.ssh/authorized_keys on that node yourself, which is what we did in the lab. cephadm now places monitors on all three hosts by itself.

Before creating OSDs, lower their memory target to fit small servers. The default is 4 GiB per OSD, and Ceph's hardware recommendations advise against going below 2 GB in real use; 1.5 GiB is fine for a lab:

# as root on ceph1
ceph config set osd osd_memory_target 1610612736
ceph orch daemon add osd ceph1:/dev/ceph-lab/osd
ceph orch daemon add osd ceph2:/dev/ceph-lab/osd
ceph orch daemon add osd ceph3:/dev/ceph-lab/osd
ceph osd tree

Each add prints Created osd(s) N on host 'cephX'. If one of them seems to hang, cephadm is usually still deploying a monitor on that host: wait a minute and run the same command again. That happened to us on the third node, and the second attempt worked at once. When ceph osd tree shows three OSDs up, the cluster is complete.

Step 5: block storage with RBD

# as root on ceph1
ceph osd pool create rbd
rbd pool init rbd
rbd create --size 2G rbd/lab-disk

Now the problem the official guides do not mention yet. Tentacle 20.2.4 is a security release that introduced a new key type, and the Linux kernel in Ubuntu 24.04 cannot use it: rbd map fails, and dmesg shows libceph: secret too big 32. The release notes for 20.2.4 warn that kernel clients may need newer kernels. The way around it is the userspace client, which works with any kernel:

# as root on ceph1
apt install -y rbd-nbd
modprobe nbd
rbd-nbd map rbd/lab-disk
mkfs.ext4 /dev/nbd0
mkdir -p /mnt/lab-disk
mount /dev/nbd0 /mnt/lab-disk
df -h /mnt/lab-disk

rbd-nbd map prints the device it created, usually /dev/nbd0. We wrote a 200 MB test file to it, and ceph df showed the replication at work: 267 MiB stored, 801 MiB used, three copies of every byte on three servers.

Step 6: a shared filesystem with CephFS

# as root on ceph1
ceph fs volume create labfs
ceph fs status labfs
apt install -y ceph-fuse
mkdir -p /mnt/labfs
ceph-fuse /mnt/labfs
echo "hello from ceph" > /mnt/labfs/hello.txt

One command creates the filesystem, its two pools and the metadata servers (one active, one standby). The kernel mount, mount -t ceph, fails on Ubuntu 24.04 with the same key problem ("adding ceph secret key to kernel failed"), so we mount with ceph-fuse, which worked first time. Any node with the ceph config and keyring can mount the same filesystem and see the same files.

Step 7: S3 object storage with RGW

# as root on ceph1
ceph orch apply rgw lab --placement="1 ceph2"
ceph orch ps --daemon-type rgw
radosgw-admin user create --uid=lab --display-name="Lab user"

The gateway listens on port 80 of ceph2. user create prints JSON with an access_key and a secret_key. Put them in ~/.s3cfg for s3cmd, with host_base and host_bucket set to ceph2:80 and use_https = False, then:

# as root on ceph1, after creating ~/.s3cfg
apt install -y s3cmd
s3cmd mb s3://lab-bucket
s3cmd put /etc/hostname s3://lab-bucket/hello.txt
s3cmd ls s3://lab-bucket

Any S3 client works the same way: point it at the gateway, use those keys. Keep port 80 closed to the internet in a lab; it is open only to the three nodes by the firewall rules in Step 1.

Step 8: pull the plug on one node

The point of Ceph is surviving failures, so test it. We powered off ceph3 without warning. Within seconds:

health: HEALTH_WARN
        Degraded data redundancy: 394/1182 objects degraded (33.333%)
mon: 3 daemons, quorum ceph1,ceph2, out of quorum: ceph3
osd: 3 osds: 2 up, 3 in

A third of the copies were missing, as expected, but nothing was lost: two monitors still formed a majority, the RBD disk accepted a new 50 MB write, and the shared file was still readable and writable. With the default replica size of 3 and minimum of 2, Ceph keeps serving as long as two copies remain. After we started ceph3 again, the boot service re-attached its OSD file, the OSD rejoined, and all 211 placement groups were back to active+clean in 50 seconds.

Lose a second node and the picture changes: with only one copy left, below the minimum of two, Ceph stops writes to protect your data until a node returns. That is why production clusters start at three nodes and grow from there.

Reading the health messages

From lab to production: what changes

• Real disks. Give every OSD a dedicated NVMe or SSD; never a file on the system disk.
• Memory. Leave osd_memory_target at 4 GiB or more, and budget about twice that per OSD in total RAM.
• Network. Ceph recommends at least 10 Gbit/s, and replication triples write traffic between nodes.
• Replicas. Keep size 3 and min_size 2. The docs warn that size 2 or min_size 1 risks data loss.
• More nodes. With five or more hosts, Ceph can rebuild the third copy elsewhere while a node is down, instead of waiting.

For a single-site cluster, fast disks and a fast network between the nodes matter more than raw CPU. If you are weighing Ceph against a full cloud platform, our comparison of Proxmox and OpenStack shows where Ceph fits in each.

Frequently asked questions

Can I run Ceph on a VPS?

Yes, for learning and testing. A KVM VPS runs cephadm and its containers fine. Because most VPS plans have one disk, use a file-backed LVM volume as the OSD, as in this guide, or a second block device if your provider offers one.

How many nodes does a Ceph cluster need?

Three is the practical minimum: with the default replica size of 3, each host holds one copy, and three monitors keep a majority when one fails. Ceph can run on one node with special settings, but then it protects you from disk failure only, not server failure.

How much RAM does a Ceph OSD need?

Ceph's default memory target is 4 GiB per OSD, and its hardware guide advises against less than 2 GB in real use. For a lab, 1.5 GiB works on 4 GB servers, as long as the cluster carries little data.

Why does rbd map fail with "secret too big 32"?

Ceph 20.2.4 introduced a new key type that older kernels, including Ubuntu 24.04's, cannot read. Use the userspace clients rbd-nbd for block devices and ceph-fuse for CephFS, or a newer kernel.

Can Ceph replace Amazon S3?

The RADOS Gateway speaks the S3 API, so most S3 tools and libraries work against it unchanged. Whether it replaces S3 depends on running it well: replication, monitoring and capacity planning become your job.

Clean up, or keep learning

When you are done, delete the three servers, or keep them and try the next experiments: add a fourth node and watch Ceph rebalance, create an erasure-coded pool, or connect a Kubernetes cluster through the Ceph CSI driver. Three KVM VPS in the same city, Amsterdam, Prishtina or Dublin, are all it takes to start.

← All articles