Ceph RBD vs CephFS: When the File System Beats Block Storage, and When It Does Not

Ceph RBD vs CephFS comes down to one question: how many machines need the same data at the same time. RBD (RADOS Block Device) hands one machine a virtual disk; CephFS hands many machines one shared POSIX file system, and pays for that with an extra daemon, the metadata server (MDS), and a round trip to it for every file you create, rename or delete. In our own lab the two were within 10% of each other on large-file reads and writes, but CephFS needed 3.85 seconds to create 5,000 small files where ext4 on RBD needed 0.50 seconds. That gap, and the sharing, decide almost every case below.

Key facts, checked on 29 October 2026:

• A CephFS file system needs "at least two RADOS pools, one for data and one for metadata", and Ceph recommends at least 3 replicas for the metadata pool on dedicated SSD or NVMe OSDs (Create a Ceph file system).
• The MDS is "single-threaded and CPU-bound for most activities", its default cache is 4 GB, and Ceph recommends at least 8 GB of RAM per MDS (Deploying metadata servers).
• RBD's exclusive lock "ensures that only a single client can write to an RBD image at any given time", but it "does not prevent two or more concurrently running clients from opening the same RBD image" (RBD exclusive locks). Two hosts with ext4 on one image still corrupt it.
• Ceph-CSI v3.18.1 (1 October 2026) serves RBD as block-mode ReadWriteOnce or ReadWriteMany and CephFS as file-mode ReadWriteOnce or ReadWriteMany (ceph-csi README). An RBD volume with a file system on it supports "ReadWriteOnce and ReadOnlyMany" only (Block devices and Kubernetes).
• Tentacle 20.2.4 is supported until an estimated 1 June 2027, Squid reaches end of life on 31 October 2026, and Umbrella 21.2.0 was released on 28 October 2026 (Ceph releases, v21.2.0 announcement). Every measurement here comes from Tentacle 20.2.4.

This completes our Ceph set: what Ceph is, RBD performance with the full fio method, RGW and S3, and the three-VPS lab. This page answers only the choice between block device and file system, and where the object gateway sits beside them.

The difference in one picture

Imagine a cabinet full of disks. RBD takes a slice of it and plugs it into one server as /dev/rbd0; the server formats it with ext4 or XFS and its own kernel keeps track of files. Ceph only sees numbered 4 MiB objects.

CephFS is the same cabinet, but Ceph itself keeps the list of files. That list lives in a separate metadata pool and is served by MDS daemons. Every client asks an MDS for directory listings, permissions and sizes, then reads and writes file contents straight to the OSDs. The docs call it "a POSIX-compliant file system built on top of Ceph's distributed object store, RADOS" and note that "no metadata state is stored locally by the MDS" (CephFS introduction): the MDS is a cache and a referee; the truth lives in RADOS.

So RBD gives one writer, no extra daemons and the lowest latency Ceph offers; CephFS gives any number of writers on any number of hosts, with an MDS to size and fail over, and a cost on every metadata operation.

RBD vs CephFS vs RGW side by side

When to use CephFS instead of RBD

CephFS wins the moment a second machine needs to write the same files. The official list of "traditional use-cases" is "shared home directories, HPC scratch space, and distributed workflow shared storage", and it has not changed in years because the need has not.

Shared uploads for a web cluster. Three WordPress or Laravel servers behind a load balancer need the same uploads/ directory. With RBD one server would own the disk and export it over NFS, a single point of failure in a cluster built to remove one. With CephFS each web server mounts the same path, an upload through server 2 is visible to server 3 at once, and the MDS fails over to its standby if its host dies.

Home directories and team shares. Per-directory quotas and snapshots are file system features; CephFS has them, and Samba or NFS can re-export the mount to Windows and macOS.

Media libraries, render farms, HPC scratch. Large files written once and read by many hosts. Here CephFS performs closest to RBD: sequential reads in our lab were 983 MiB/s on CephFS and 930 MiB/s on RBD in the same VM, because file contents go client to OSD with no MDS in the data path. Kubernetes ReadWriteMany volumes are the same pattern, covered below.

When RBD is the right answer

Everything with one writer that cares about latency. A VM disk is the obvious case: QEMU opens the image through librbd, the guest sees a normal disk, and copy-on-write clones let OpenStack or CloudStack create a VM from a template in seconds (RBD snapshots and layering).

Databases belong on RBD too, even when it feels convenient to put PostgreSQL on the shared file system next to the application. A database creates, renames and fsyncs files constantly; on RBD those operations happen in the client's own kernel, on CephFS each one involves the MDS. The single-writer rule also protects you: a database two hosts could open at once is a database waiting to be corrupted. The same goes for message queues, build caches that belong to one runner, and any application written for a local disk.

What breaks when you choose wrong

A database on CephFS

It works, then it disappoints. Our single-queue 4 KiB write latency was 1.86 ms on CephFS and 1.99 ms on RBD, so data writes are not the problem; the 7.7 and 21 times slower file create and delete in the table below are, and a MySQL with hundreds of tables lives inside those numbers all day.

Millions of small files and MDS pressure

The MDS caches recently used inodes (the records describing each file) up to mds_cache_memory_limit, 4 GiB by default. When the files clients actively touch outgrow it, the MDS asks clients to hand back capabilities, and if they are slow you see "Client failing to respond to cache pressure" and "Too many inodes in cache" (CephFS health messages). The troubleshooting guide names the usual cause: "an active file set that is larger than the MDS cache" (CephFS troubleshooting). The fixes are more MDS RAM, several active MDS daemons, or fewer files per directory; for huge flat collections of whole files, the docs point to the object gateway instead (application best practices).

One RBD image on two hosts

The exclusive lock passes between clients "in a cooperative fashion", so both hosts can map the image and both can write, just not at the same instant. Ext4 and XFS assume they alone touch the disk, the second host's kernel has a stale picture of free blocks, and the file system is corrupted. Sharing one block device safely needs a cluster file system such as GFS2 or OCFS2 with its own lock manager, which is exactly the complexity CephFS exists to remove. What the lock does give you is fencing: a new client that finds a dead holder "not only breaks the lock but also blocklists the previous lock holder", "a form of storage-level resource fencing" in the docs' words.

One trap on the CephFS side: a shared writable mmap is not kept coherent between hosts, and "CephFS does not currently maintain the atime field" (differences from POSIX).

RBD vs CephFS performance: what our lab showed

Measured on 11 October 2026 in one Tentacle 20.2.4 VM with 2 vCPU and three OSDs, and published with the fio command lines and caveats in the RBD performance article. They show the relative cost of each layer, not what a production cluster delivers.

The lesson is not "CephFS is slow". For bytes it is the same storage, striped over the same OSDs. The lesson is that CephFS turns file creation into a network protocol with a daemon in the middle, so a workload is fast or slow according to how many metadata operations it does per useful byte. Test with the number of clients you will have; one client says nothing about the MDS under ten.

Ceph storage classes in Kubernetes: cephfs for RWX, rbd for RWO

Ceph-CSI ships two drivers, and most clusters define one StorageClass for each. For the RBD driver (rbd.csi.ceph.com) the docs state the rule directly: volumeMode: Filesystem "can support both ReadWriteOnce and ReadOnlyMany accessMode claims", while volumeMode: Block "can support ReadWriteOnce, ReadWriteMany, and ReadOnlyMany". Block-mode RWX exists for software like KubeVirt live migration, where two nodes briefly hold one disk and the hypervisor coordinates the hand-over; it is not a way to share files between ordinary pods.

The CephFS driver creates a subvolume per claim. Rook, the usual way to run Ceph inside Kubernetes, puts it as "A filesystem storage (also named shared filesystem) can be mounted with read/write permission from multiple pods"; its example uses a replicated metadata pool of size 3, one active MDS with a standby, and a ReadWriteMany claim (Rook shared file system, read 29 October 2026).

On OKD or OpenShift these classes slot into the platform as described in our OKD scalability article; the block storage comparison puts RBD next to LINSTOR, Longhorn and ZFS for clusters that do not want Ceph's footprint.

Hardware, pools and MDS sizing for CephFS

RBD needs nothing beyond the cluster you already have. CephFS adds three decisions.

The metadata pool

It is small, usually a few gigabytes, and hit on every file operation, so it should be the fastest thing in the cluster: "We strongly suggest that the CephFS metadata pool be provisioned on dedicated SSD / NVMe OSDs", with "at least 3 replicas" and a small PG count, 64 or 128 in large clusters. The hardware guide tells you to build a CRUSH rule that includes only SSD media for it (hardware recommendations). Erasure coding is for the data pool, never the metadata pool. One command created both pools and the MDS daemons in our three-node lab:

# as root on a cluster admin node (from our three-VPS lab; needs a running cluster)
ceph fs volume create labfs
ceph fs status labfs

It produced cephfs.labfs.meta, cephfs.labfs.data and two MDS daemons. On real hardware, create the metadata pool with an SSD-only CRUSH rule first and pass it with --meta-pool.

MDS RAM and CPU

The MDS "under the most aggressive client loads uses about 2 to 3 CPU cores" and wants a fast clock more than many cores. Its default cache is 4 GiB, Ceph recommends "at least 8GB of RAM for the MDS to support this cache size", and "an MDS serving a large cluster of clients (1000 or more) will use at least 64GB of cache". The limit is a target, not a wall; a health warning fires at 150% of it (MDS cache configuration). How many files fit? Current docs size by memory, but when Ceph introduced the memory limit it wrote that "In the past, we have suggested 100k inodes per 1GB of RAM" (Ceph blog, Luminous). By that rule a million actively used inodes want about 10 GB of cache, so an MDS host with 16 GB of RAM; treat it as a starting point and watch ceph fs status under real load.

One MDS or several

"Each CephFS file system is configured for a single active MDS daemon by default." Add active daemons when metadata throughput is the limit, typically with "many clients, especially ones working in separate directories"; one application on one client "will not benefit" (multiple active MDS):

# as root on a cluster admin node (official docs; not run in our labs)
ceph fs set labfs max_mds 2
setfattr -n ceph.dir.pin -v 1 /mnt/cephfs/projects/team-b

The second line pins a directory tree to MDS rank 1 so two busy teams do not share one daemon. Keep a standby beyond max_mds on a different host; otherwise "a hardware failure on a single node may result in the file system becoming unavailable".

Kernel client or ceph-fuse, and the new mount syntax

The kernel driver gives better performance; ceph-fuse mounts are "less performant than kernel driver mounts, but they are easier to manage and easier to upgrade" (mount using FUSE). The practical reason to keep ceph-fuse installed is that it tracks the cluster's Ceph version while the kernel driver tracks your kernel. We hit that in the three-VPS lab: Tentacle 20.2.4 introduced a key type Ubuntu 24.04's kernel could not use, mount -t ceph failed, and ceph-fuse worked at once. Distribution packages lag too: on a 1 GiB Debian 13 container we started on 29 October 2026 with no cluster attached, apt-get install ceph-common ceph-fuse gave ceph version 18.2.7 (...) reef (stable), two releases behind Tentacle, with /sbin/mount.ceph and an rbd device map that offers --device-type nbd as the user-space fallback and --exclusive to turn the cooperative lock into a hard one for a disk that must never be mapped twice.

The kernel mount takes a device string of the form user@fsid.fsname=/path; the fsid can be left out when ceph.conf is present, but the dot "must be a part of the device string" (mount using the kernel driver):

# as root on a client with /etc/ceph/ceph.conf and the user's keyring
mount -t ceph cephuser@.cephfs=/ /mnt/cephfs -o secretfile=/etc/ceph/cephuser.secret
# the same mount in /etc/fstab
cephuser@.cephfs=/   /mnt/cephfs   ceph   mon_addr=192.168.0.1:6789,noatime,_netdev   0 0

We used this form in the single-VM lab (mount -t ceph fsuser@.labfs=/ /mnt/cephfs). The client user comes from ceph fs authorize <fs> client.<name> <path> rw, which confines it to a path; add p for quotas and s for snapshots (CephFS client capabilities).

Snapshots, quotas and backups on each side

RBD snapshots are per image and crash-consistent only: RBD does not know what file system is inside, so the docs "recommend that you pause or stop I/O before taking a snapshot", with fsfreeze on a mounted file system. A protected snapshot can be cloned copy-on-write, and rbd flatten later cuts the clone loose:

# as root on a cluster admin node (official docs; not run in our labs)
rbd snap create rbd/vm-disk@before-upgrade
rbd snap protect rbd/vm-disk@before-upgrade
rbd clone rbd/vm-disk@before-upgrade rbd/vm-disk-test

RBD backups are block backups: rbd export-diff between two snapshots gives incremental copies.

CephFS snapshots are per directory, enabled by default on new file systems, and made with a plain mkdir inside the hidden .snap directory (CephFS snapshots); because the snapshot covers the tree, it is consistent for files, which an RBD snapshot cannot promise. The snap_schedule module automates it, keeping at most 50 snapshots per directory (snapshot scheduling). Quotas are extended attributes on any directory; they are "cooperative and non-adversarial", enforced by the client, and writers are "stopped within 10s of seconds of crossing the configured limit", so they are a budget, not a wall against hostile tenants (CephFS quotas):

# as root on a cluster admin node, then on a CephFS client with the p capability (official docs; not run in our labs)
ceph mgr module enable snap_schedule
ceph fs snap-schedule add /uploads 1h
ceph fs snap-schedule retention add /uploads 24h4w
setfattr -n ceph.quota.max_bytes -v 50Gi /mnt/cephfs/uploads
setfattr -n ceph.quota.max_files -v 1000000 /mnt/cephfs/uploads

Kubernetes and OpenStack Manila get the same features packaged as subvolumes: ceph fs subvolume create <vol> <name> --size <bytes> sets the quota, and ceph fs subvolume snapshot create and snapshot clone handle copies (FS volumes and subvolumes). CephFS backups are file backups: mount a snapshot and run restic or rsync against it, or use cephfs-mirror to a second cluster. A snapshot on the same cluster is not a backup, on either side.

Decision table: workload to RBD, CephFS or RGW

When in doubt: if you can name the one server that writes, RBD. If you cannot and the data is files, CephFS. If the data is files an application uploads and fetches whole, RGW.

Where RS Computers fits: a lab, not a production cluster

Ceph in production is a cluster of servers with dedicated disks, 10 Gb/s between them and plenty of RAM. It is not a feature you switch on inside a single VPS, and we do not sell it as one. What a VDS is good for is learning the difference this article describes before you spend on hardware: build the three-node lab, put an RBD image and a CephFS mount on it, and run the small-file test from the performance article against your own application.

Our single-VM lab ran out of memory at 4 GiB the moment a CephFS was created on top of three OSDs, which is why VDS Small with 8 GB is the smallest plan we name, and why VDS plans with dedicated vCPUs suit Ceph better than shared-vCPU VPS plans: OSDs and an MDS want CPU time when they want it. Every VDS has a 10 Gb/s port, the speed Ceph asks for between nodes. Compare them on the VPS and VDS plans page.

Frequently asked questions

What is the difference between Ceph RBD and CephFS?

RBD is a virtual block device: one machine maps it, formats it and is its only writer. CephFS is a shared POSIX file system: many machines mount the same directory tree, and Ceph's metadata servers (MDS) keep the list of files consistent between them. Both store data as replicated objects on the same OSDs.

Is CephFS slower than RBD?

For large files, no: in our Tentacle 20.2.4 lab the two were within 10% on random 4K and sequential 1M tests. For creating, listing and deleting many small files, yes: 5,000 small files took 3.85 s to create on CephFS and 0.50 s on ext4 over RBD, because each file operation is a request to the MDS.

When should I use CephFS instead of RBD?

When more than one server or pod must read and write the same files at the same time: shared upload directories behind a load balancer, home directories, media libraries, HPC scratch and Kubernetes ReadWriteMany volumes. If one machine is the only writer, RBD is simpler and has no MDS to run.

Can I run a database on CephFS?

It runs, but RBD is the better home. A database creates, renames and fsyncs files constantly, and on CephFS each of those operations travels to the MDS. Put PostgreSQL or MySQL on an RBD volume with ext4 or XFS and keep CephFS for the files the application shares.

Can two servers mount the same RBD image?

They can map it, because the exclusive lock passes between clients, but with ext4 or XFS on top the second writer corrupts the file system. Sharing a block device safely needs a cluster file system with its own lock manager; in Ceph the simple answer is CephFS.

Which Kubernetes access mode do RBD and CephFS support?

With Ceph-CSI, an RBD volume formatted with a file system is ReadWriteOnce or ReadOnlyMany; raw block-mode RBD can be ReadWriteMany for software that coordinates its own writes, such as VM live migration. CephFS volumes support ReadWriteMany, so a Deployment with several replicas that write files needs the CephFS StorageClass.

Build the lab before you buy the racks

Three VDS servers in one city are enough to watch an MDS fail over, share a CephFS directory between nodes and run your own application against both. Pick the plans on the VPS and VDS page, follow the three-node guide, and if you want a second opinion on whether your workload belongs on a block device or a file system, describe it to us on Telegram.

← All articles