Every object you upload to enum Object Storage is split into six fragments: four hold your data, two hold parity. Each fragment lands on a different server. Any four of the six are enough to read the object back.
That scheme is called erasure coding, and “4+2” is shorthand for it. This post covers how it works on our cluster, what it protects you from, where its limits are, and why we chose it.
The setup
The storage cluster runs Ceph with its S3 gateway, on six storage servers in our Frankfurt data centre (NTT FRA1). Each server has:
- 4x 12 TB HDDs for object data
- 2x 2 TB NVMe drives, each shared by two of the HDDs
- 2x 10G uplinks, bonded
The data pool is erasure coded with 4 data chunks and 2 coding chunks, and the failure domain is the server. Ceph’s placement rules won’t put two fragments of the same object on one machine. With six servers and six fragments, every object touches every server exactly once.
The bucket index and other gateway metadata live in separate pools that keep three full copies instead of using erasure coding. Those pools are small and see many tiny writes, which suits replication better than parity.
How 4+2 works
Ceph doesn’t store a large object as one piece. The S3 gateway cuts it into stripes of a few megabytes, and each stripe is then encoded on its own:
- The stripe is split into four equal data chunks.
- Ceph computes two parity chunks from those four.
- All six chunks go to six different servers.
Reading a stripe needs any four of the six chunks. If all four data chunks are there, Ceph reads them directly. If one or two are missing, it recomputes them from the parity. The math is Reed-Solomon, the same family of codes used on CDs and QR codes.
1.5x instead of 3x
The usual alternative is replication: keep three full copies on three servers. Three copies also survive losing two of them, so the protection against failed servers is the same. The cost isn’t:
| 3 copies | EC 4+2 | |
|---|---|---|
| Disk per TB | 3 TB | 1.5 TB |
| Survives | 2 servers | 2 servers |
| Usable | 33% | 67% |
Storage is mostly disks, so half the disks for the same fault tolerance is the biggest lever we have on price. It is why object storage costs €0.02 per GB a month and API calls are free.
Erasure coding has costs of its own. Every write touches six servers instead of three, and computing parity uses CPU. Small objects pay more overhead, because a 4 KB object still becomes six chunks. Rebuilding a lost chunk means reading four others over the network, where replication just copies one. For object storage, with mostly large, write-once objects, those costs are worth paying.
HDDs for data, NVMe for metadata
HDDs are cheap per terabyte and fine at reading and writing large chunks in sequence. They are slow at the small random writes that come with object storage: bucket index updates, object metadata, the storage engine’s own write-ahead log.
So we split the work. Each HDD is paired with a slice of an NVMe drive. Ceph’s storage engine, BlueStore, keeps its database and write-ahead log on the NVMe, and that database is where the bucket index and object metadata end up. The HDD only gets the bulk data.
Listing a bucket or checking an object’s metadata runs on flash. Streaming a 2 GB backup runs on spinning disks that are good at exactly that.
Every disk is encrypted at the device level (dm-crypt, AES-256), so a drive that leaves the data centre is unreadable.
What it survives
A failed disk. The fragments on that disk are rebuilt on the other three disks in the same server. The one-fragment-per-server rule still holds, and Ceph starts the rebuild without anyone touching the machine. While it runs, every object stays readable.
A failed server. Every object has lost one of its six fragments, so five are left. Reads and writes carry on. We replace or repair the server, and Ceph fills its fragments back in.
A second failure while the first is being fixed. With two servers or two disks gone, four fragments remain, and four is enough to rebuild every object. No data is lost. In this state Ceph pauses access to the affected data instead of writing new objects with zero spare fragments. Access resumes as soon as a fifth fragment is back.
What it doesn’t survive, and why we say so
A failed server isn’t rebuilt elsewhere. Six servers for six fragments means there is no seventh machine to move a dead server’s fragments to. Until we bring it back, objects run with one fragment of headroom instead of two. A seventh server would let Ceph rebuild onto the others without waiting for hardware. We picked six to start, and replacing a server quickly is part of our on-call job.
One rack, one site. All six servers sit in one rack in one data centre. Erasure coding protects against disks and servers failing. It does not protect against something that takes out the rack or the building. There is no cross-region replication yet, and our guide to S3 storage in Europe says the same: the site is the failure boundary.
You deleting things. Erasure coding faithfully stores whatever you tell it to, including a delete or an overwrite with garbage. That is what versioning and Object Lock are for.
For most workloads this means: treat enum Object Storage as your primary copy, and keep a second copy of anything you can’t afford to lose somewhere else. That applies to any single-site storage, ours included.
Why not 8+3, or 6+3?
Wider schemes cost less disk per terabyte. 8+3 has 1.375x overhead and survives three failures. But every fragment needs its own server, so 8+3 needs at least eleven. With six servers, 4+2 is the widest scheme that keeps one fragment per server and still survives two failures. As the cluster grows, the trade-offs change, and we’ll write about it when they do.
Try it
enum Object Storage is S3 compatible, with versioning, lifecycle rules and Object Lock. Create a bucket in the console or with enumctl, and see the docs for the details. Questions about how your data is stored? Write to mail@enum.co.