Ceph Object Storage: Build Your Own S3 with RGW, Step by Step (Lab-Tested)

Ceph object storage is the RADOS Gateway (RGW): a web service that speaks the Amazon S3 and OpenStack Swift APIs and keeps every bucket and file inside a Ceph cluster. Your apps talk to it exactly as they would talk to S3, with an endpoint URL, an access key and a secret key, while Ceph spreads the data over many disks and servers underneath. We built one on a single test VM with Ceph Tentacle 20.2.4, created a bucket, a lifecycle rule, a bucket policy and a presigned link, and timed a 1 GB upload. Every command below ran in that lab, including two failures you will probably hit too.

Key facts, checked on 11 October 2026:

• Ceph Tentacle 20.2.4, released on 19 August 2026, is the current stable release; Squid 19.2 reaches its estimated end of life on 31 October 2026 (Ceph active releases).
• RGW is "an object storage interface built on top of librados", and its S3 and Swift APIs "share a common namespace", so a file uploaded with one can be read with the other (Ceph Object Gateway docs).
• The MinIO GitHub repository was archived on 25 April 2026 and says the community edition "is now distributed as source code only" (github.com/minio/minio).
• In our single-VM lab, the whole cluster (monitor, two managers, three OSDs and one gateway) used 1,578 MB of RAM, and a 1 GiB file uploaded in 10.4 to 15.8 seconds.
• An erasure-coded 2+1 pool stored 3.0 GiB of objects in 4.5 GiB of raw disk, exactly the 1.5x overhead the maths predicts.

If you want the bigger picture first (block storage, CephFS, OpenStack and Kubernetes), read our Ceph block and object storage overview. If you want a real three-server cluster, follow the three-VPS Ceph test lab. This page stays on one job: S3 storage with Ceph.

How Ceph object storage works, layer by layer

Think of Ceph object storage as a stack. Your app sits at the top and only ever sees S3. Everything below it is Ceph's business.

"Object storage device" means a disk daemon, not an S3 object

This trips up many people searching for "ceph object storage device". OSD stands for object storage daemon (older papers say device), and it is the process that owns one physical disk. An S3 object is a file you upload. RGW cuts each uploaded file into RADOS objects, by default up to 4 MiB each, and CRUSH places them on OSDs. So one 1 GB upload becomes a few hundred RADOS objects spread across many OSDs, and a cluster with 12 disks has 12 OSDs no matter how many files it holds.

Users, keys and buckets

RGW keeps its own user list. radosgw-admin user create makes a user and prints an access key and a secret key, the same pair AWS gives you. Each user can own up to 1,000 buckets by default (we saw "max_buckets": 1000 in the output). A bucket is a flat container of objects; the "folders" you see in tools are only key prefixes such as public/. On top of that, RGW supports bucket policies, lifecycle rules, versioning, tagging and website hosting, according to its S3 API feature table.

Replication or erasure coding: the space maths for buckets

Replication keeps full copies. Erasure coding (EC) cuts data into k pieces and adds m parity pieces, so any m pieces can be lost. EC saves a lot of disk, and it suits RGW data pools well because S3 objects are written once and read many times. The bucket index pool should stay replicated, because it is small and busy.

The formula is (k+m)/k. EC 2+1 and EC 4+2 cost the same disk, but 4+2 survives two failures, so it is the usual choice once you have six or more servers. Ceph's erasure code guide covers the profiles.

Multisite in one paragraph

RGW can copy buckets between clusters in different cities. You define a realm, which holds zonegroups, which hold zones; each zone is one Ceph cluster with its own gateways, and zones in the same zonegroup sync objects to each other asynchronously, so both sides can take writes. It is how you keep a copy in Amsterdam and another in Dublin. One Tentacle detail: the help text of the rgw_sigv4_insecure option in 20.2.4 says multisite users must set it until every cluster is upgraded, then turn it off again. The setup steps are in the multisite docs; we did not test multisite in this lab.

The lab: Ceph RGW on one VM, step by step

Our lab ran on an Amsterdam test server with Incus. We created one KVM virtual machine with Ubuntu 24.04, 2 vCPU and 4 GiB of RAM, plus three empty 8 GiB disks for the OSDs, and a separate Debian 13 container as the S3 client. One VM with three disks is a teaching cluster: it protects you from a dead disk, not from a dead server. On a VPS or VDS, extra block devices would play the role of our three disks.

# as root on the Incus host (only if you build the lab the way we did)
incus init images:ubuntu/24.04 wf-rgw --vm -c limits.memory=4GiB -c limits.cpu=2 -d root,size=25GiB
for n in 1 2 3; do
  incus storage volume create default wf-rgw-osd$n --type=block size=8GiB
  incus storage volume attach default wf-rgw-osd$n wf-rgw
done
incus start wf-rgw

Inside the VM, lsblk showed the 25 GB system disk as sda and the three empty disks as sdb, sdc and sdd.

Step 1: install cephadm

# as root on the VM
hostnamectl set-hostname ceph1
echo "10.141.188.211 ceph1" >> /etc/hosts
apt update
apt install -y podman lvm2 chrony curl openssh-server
systemctl enable --now ssh
curl --silent --remote-name --location https://download.ceph.com/rpm-tentacle/el9/noarch/cephadm
chmod +x cephadm
./cephadm add-repo --release tentacle
./cephadm install

Replace 10.141.188.211 with your server's own address. Do not skip openssh-server: the Ubuntu image we used had no SSH server, and our first bootstrap failed after 1 minute 38 seconds with Connect call failed ('10.141.188.211', 22), because cephadm manages even the local host over SSH. The official guide then installs the ceph-common package; download.ceph.com sent it to us at about 10 KB/s, so we stopped and used cephadm shell, which runs the same ceph and radosgw-admin commands from the container image.

Step 2: bootstrap a single-host cluster

# as root on the VM
cephadm bootstrap --mon-ip 10.141.188.211 --single-host-defaults --skip-monitoring-stack
cephadm shell -- ceph --version

Expected output ends with Bootstrap complete., a dashboard address on port 8443 and a one-time password. On the second try it took 45 seconds. The version line read ceph version 20.2.4 (7f793731...) tentacle (stable). According to the cephadm install guide, --single-host-defaults sets three things: copies may share a host (osd_crush_chooseleaf_type = 0), pools keep 2 copies instead of 3, and the standby manager does not run modules. The same guide says such clusters "are generally not suitable for production". --skip-monitoring-stack leaves out Prometheus and Grafana to save memory.

Step 3: turn the empty disks into OSDs

# as root on the VM
cephadm shell -- ceph config set osd osd_memory_target 939524096
cephadm shell -- ceph orch device ls
cephadm shell -- ceph orch apply osd --all-available-devices
cephadm shell -- ceph -s

The first line caps each OSD at 896 MiB, because the default target of 4 GiB per OSD would not fit three OSDs into a 4 GiB machine. ceph orch device ls listed sdb, sdc and sdd as 8192M, available. About 35 seconds after the apply, ceph -s showed HEALTH_OK and osd: 3 osds: 3 up, 3 in with 24 GiB of raw space.

Step 4: an erasure-coded data pool, then the gateway

RGW creates its pools by itself, all replicated. To store bucket data with erasure coding, create the data pool before the gateway starts. RGW then uses it as it is.

# as root on the VM
cephadm shell -- ceph osd erasure-code-profile set ec21 k=2 m=1 crush-failure-domain=osd
cephadm shell -- ceph osd pool create default.rgw.buckets.data erasure ec21
cephadm shell -- ceph osd pool application enable default.rgw.buckets.data rgw
cephadm shell -- ceph orch apply rgw lab --placement="1 ceph1" --port=8080
cephadm shell -- ceph orch ps --daemon-type rgw
curl http://127.0.0.1:8080

crush-failure-domain=osd lets the three pieces sit on three disks of one host; on a real cluster use host. The profile showed plugin=isa, the Intel ISA-L library that Tentacle picks by default. After about 40 seconds, ceph orch ps listed rgw.lab.ceph1... *:8080 running using 80 MB, and the anonymous curl returned an empty ListAllMyBucketsResult, which means the gateway answers. Without --port, RGW listens on port 80 (RGW service docs).

Step 5: create an S3 user

# as root on the VM
cephadm shell -- radosgw-admin user create --uid=dev --display-name="Dev user"

The JSON output contains the part you need:

# output (keys shortened)
"keys": [
    {
        "user": "dev",
        "access_key": "FV9W11XV...",
        "secret_key": "5K8c5P9Z...",
        "active": true
    }
],

Treat the secret key like a password.

Step 6: use it from another machine with the AWS CLI

The client was a Debian 13 container with Debian's own awscli 2.23.6 and s3cmd 2.4.0 packages, used by a normal user called dev. Two small files point the AWS CLI at the gateway:

# as the dev user on the client
mkdir -p ~/.aws
cat > ~/.aws/credentials <<'EOF'
[rgw]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY
EOF
cat > ~/.aws/config <<'EOF'
[profile rgw]
region = default
endpoint_url = http://10.141.188.211:8080
EOF
export AWS_PROFILE=rgw
aws s3 mb s3://lab-bucket
echo "hello from ceph rgw" > hello.txt
aws s3 cp hello.txt s3://lab-bucket/
aws s3 ls s3://lab-bucket/

Expected: make_bucket: lab-bucket, then upload: ./hello.txt to s3://lab-bucket/hello.txt, then a listing line with the date and 20 hello.txt.

Step 7: a lifecycle rule, a bucket policy and a presigned link

A lifecycle rule deletes old objects for you. This one removes anything under tmp/ after 7 days:

# as the dev user on the client
cat > lifecycle.json <<'EOF'
{
  "Rules": [
    {
      "ID": "expire-tmp",
      "Filter": { "Prefix": "tmp/" },
      "Status": "Enabled",
      "Expiration": { "Days": 7 }
    }
  ]
}
EOF
aws s3api put-bucket-lifecycle-configuration --bucket lab-bucket --lifecycle-configuration file://lifecycle.json
aws s3api get-bucket-lifecycle-configuration --bucket lab-bucket

The get command printed the rule back. RGW processes lifecycle rules in a nightly window, 00:00 to 06:00 by default (rgw_lifecycle_work_time), so do not expect objects to vanish the moment they are 7 days old.

A bucket policy grants access. This one makes only the public/ prefix readable by anyone:

# as the dev user on the client
cat > policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadPublicFolder",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::lab-bucket/public/*"
    }
  ]
}
EOF
aws s3api put-bucket-policy --bucket lab-bucket --policy file://policy.json
aws s3 cp hello.txt s3://lab-bucket/public/hello.txt
curl -s -o /dev/null -w "%{http_code}\n" http://10.141.188.211:8080/lab-bucket/public/hello.txt
curl -s -o /dev/null -w "%{http_code}\n" http://10.141.188.211:8080/lab-bucket/hello.txt

The first curl printed 200 and the second 403: the policy opened one prefix and nothing else. For a private file you want to share for a short time, use a presigned URL instead:

# as the dev user on the client
URL=$(aws s3 presign s3://lab-bucket/hello.txt --expires-in 600)
curl -s "$URL"

The URL carries X-Amz-Expires=600 and a signature, and curl printed hello from ceph rgw without any keys. After 10 minutes the link stops working. We also turned on versioning with aws s3api put-bucket-versioning --bucket lab-bucket --versioning-configuration Status=Enabled; uploading a new file under the same key left two versions, which aws s3api list-object-versions showed. s3cmd worked too, with host_base and host_bucket set to the gateway address and use_https = False in ~/.s3cfg.

Step 8: how fast, and how much disk

# as the dev user on the client
head -c 1G /dev/urandom > big.bin
time aws s3 cp big.bin s3://lab-bucket/big.bin
time aws s3 cp s3://lab-bucket/big.bin back.bin
cmp big.bin back.bin && echo identical

Read these numbers as a lab result, not a benchmark. Two vCPUs ran the gateway, three OSDs and the erasure coding maths, and all three virtual disks lived on the same physical drive. Versioning explains the 3.0 GiB: three uploads of the same key kept three versions, so plan lifecycle rules for old versions when you turn it on. For CPU and disk tests of a server before you build on it, see our VPS benchmark commands.

The restic surprise: Access Denied on Tentacle 20.2.4

Restic is a natural tenant for an RGW bucket, so we tried it. With restic 0.18.0 and its built-in s3: backend, restic init failed with client.PutObject: Access Denied. RGW's debug log gave the reason: Signature rejected: 'content-type' supplied but not in CanonicalHeaders. Ceph 20.2.4 fixed CVE-2026-54330, a SigV4 verifier error that let a client attach extra x-amz-* headers and escalate privileges (Tentacle release notes), and in our lab the stricter check also rejected requests from the S3 library inside restic, which sends a Content-Type header without signing it. The AWS CLI and s3cmd were not affected.

Setting rgw_sigv4_insecure to true made restic work, but that switch reopens the hole the update closed. The clean fix was to let rclone carry the traffic, which restic supports natively:

# as the dev user on the client (after apt install restic rclone as root)
mkdir -p ~/.config/rclone
cat > ~/.config/rclone/rclone.conf <<'EOF'
[rgw]
type = s3
provider = Ceph
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
endpoint = http://10.141.188.211:8080
EOF
export RESTIC_PASSWORD='choose-a-long-password'
restic -r rclone:rgw:restic-rclone init
restic -r rclone:rgw:restic-rclone backup /etc/apt

Expected: created restic repository ... at rclone:rgw:restic-rclone, then snapshot ... saved. If other S3 tools return 403 after a Ceph upgrade, check the RGW log for the same message before you start changing keys.

Ceph RGW vs MinIO vs Garage (and SeaweedFS)

Ceph is the heavy option. It gives you object, block and file storage from one cluster, but it wants several servers and someone who understands it. Here is how the main self-hosted S3 servers compare, checked on 11 October 2026:

Sources: MinIO repository, Garage home page and S3 compatibility table, Garage mirror (licence), SeaweedFS repository and releases.

A quick rule: one server and a handful of buckets, pick Garage or SeaweedFS. Several servers, a need for policies and versioning, or VM disks from the same hardware, pick Ceph.

From lab to production on VDS nodes

• Start with three servers or more, each with its own OSD disks, and keep the default of 3 copies for metadata and index pools. Use the three-node lab as the template.
• Give the network room. Every write crosses it several times, and Ceph recommends at least 10 Gb/s. Our VDS Small (4 vCPU, 8 GB RAM, 240 GB NVMe), VDS Medium (8 vCPU, 16 GB, 480 GB) and VDS Large (16 vCPU, 32 GB, 960 GB) plans come with 10 Gb/s ports, own IPv4 and IPv6, and unmetered traffic.
• Budget memory: the default OSD target is 4 GiB per OSD, plus the gateway, monitors and managers.
• Run two or more RGW daemons behind a load balancer, and put HTTPS in front of them. Our guide to Caddy, Nginx and Traefik reverse proxies covers certificates.
• Never expose port 8080 or the dashboard on 8443 to the internet without TLS and a firewall.

Keep all nodes of one cluster in the same city: Amsterdam, Dublin or Prishtina. Use multisite, not one stretched cluster, to keep a second copy elsewhere.

Frequently asked questions

What is Ceph object storage?

Ceph object storage is the Ceph Object Gateway, called RGW or radosgw. It is an HTTP service that offers the Amazon S3 and OpenStack Swift APIs and stores the data in a Ceph cluster, which spreads it over many disks and servers with replication or erasure coding.

Is Ceph RGW compatible with Amazon S3?

Yes, for most everyday work. In our lab the AWS CLI, s3cmd and rclone created buckets, uploaded files, set lifecycle rules and bucket policies and produced presigned URLs against Ceph 20.2.4 without changes. Some newer or niche S3 features differ, so check Ceph's S3 feature table for anything unusual.

What is an OSD in Ceph?

An OSD (object storage daemon) is the Ceph process that manages one disk, stores data on it and copies data to other OSDs. It is not an S3 object. A cluster usually runs one OSD per disk, so 12 disks means 12 OSDs.

How many servers does Ceph object storage need?

Three servers is the practical minimum for production, so that each copy of the data lives on a different machine. One server works for learning with cephadm bootstrap --single-host-defaults, as in our lab, but it only survives a disk failure, not a server failure.

How is Ceph used in cloud computing?

Cloud platforms use Ceph as their storage layer: RBD for virtual machine disks, RGW for S3 or Swift buckets, and CephFS for shared files. OpenStack and Kubernetes (through Rook and Ceph CSI) both integrate with it, so one cluster can serve every storage need of a private cloud.

Should I use Ceph or MinIO in 2026?

MinIO's community repository was archived on 25 April 2026 and is now source only, with the vendor pointing users to its AIStor editions. Ceph RGW is actively released, with 20.2.4 in August 2026, but needs at least three servers to make sense. For a single small server, Garage or SeaweedFS are lighter choices.

Build your S3 cluster on our servers

A Ceph lab fits on one server; a real Ceph object store wants three or more VDS nodes with 10 Gb/s in the same city. Compare the VPS and VDS plans and see availability by city on the plans page, or ask us on Telegram which plan fits your cluster. If you would like us to install it for you, we quote setup work per job.

← All articles