How Object Storage Reaches Eleven Nines: Index, Erasure Coding and Self-Healing
What happens when you PUT an object: routing, authorisation, a sharded metadata index, erasure-coded placement across zones, and background auditors.
On this page
The first object store launched with a promise: put any amount of data in, get it back later. Nearly two decades on, it holds hundreds of trillions of objects. Knowing the machinery behind a simple PUT explains most of its quirks, and most of the performance advice.
What a write actually does
- The client sends a PUT
Through an SDK, CLI or REST API. Large objects use multipart upload: split into parts, upload in parallel, retry parts independently, then complete the upload so parts become one object.
- DNS and routing
The bucket’s hostname resolves to a nearby entry point. Features such as multi-region access points can route to the lowest-latency region.
- Front-end servers check the request
Identity and permissions are verified (IAM, bucket policy), headers and signatures validated, and the request goes to a front-end server chosen by load balancing. An unauthorised request ends here with a 403.
- The key is recorded in the index
A metadata store maps the key to where its bytes live, along with size, version and permissions.
- Data is placed with erasure coding
The object is split into fragments with computed parity, spread over many disks and several availability zones.
- The client gets a confirmation
Including an
ETag(a checksum) and, with versioning enabled, aVersionId. Reads after this see the new object: modern S3 provides strong read-after-write consistency.
The index: why prefixes matter
Looking up a key must not scan billions of entries. The metadata index is partitioned by key range (lexicographic order), so each partition handles a slice of the keyspace. Two things follow.
- Requests spread across many partitions run in parallel, which is how request rates reach very high levels.
- If most of your traffic shares one prefix, such as every key starting with the same date, those requests pile onto one partition. The service rebalances partitions as load grows, but a sudden, concentrated burst can still hit throttling first.
Put high-cardinality characters early in the key (a hash or short random prefix, a customer id) and spread load across many prefixes. Remember per-prefix request limits scale with the number of prefixes you use, so parallel prefixes mean parallel capacity.
hot: logs/2026-11-14/00001.json logs/2026-11-14/00002.json (one prefix, one partition)
spread: 3f/logs/2026-11-14/00001.json a9/logs/2026-11-14/00002.json
Erasure coding in one picture
Keeping three full copies tolerates two failures at 200 percent overhead. Erasure coding gets similar or better resilience for less storage.
data split into k fragments + m parity fragments = k + m pieces
any k of the k + m pieces can rebuild the object
example: k = 6, m = 3 -> survive any 3 lost pieces, overhead 50%
| Full replication (3 copies) | Erasure coding (6+3) | |
|---|---|---|
| Failures tolerated | 2 | 3 |
| Storage overhead | 200% | 50% |
| Read cost | Read one copy | Read k pieces, reassemble |
| Repair cost | Copy a surviving replica | Read k pieces to rebuild one |
The price is compute and network work on rebuild. Placing pieces across zones means losing a whole zone is just losing a few pieces.
Where durability really comes from
Eleven nines is not one clever trick. It is redundancy plus vigilance.
Durability and availability are different promises. An object can be perfectly safe and temporarily unreachable.
Handling a hundred million requests a second
No single machine does this. The system relies on:
- Parallel index partitions, each serving a range of keys.
- Caching for hot metadata and data paths.
- Dynamic rebalancing, splitting partitions that run hot.
- Latency-aware routing to steer requests to healthy, fast paths.
- Many small services, each scaled and deployed independently, so one component’s trouble stays local.
What to do as a user
| Practice | Why |
|---|---|
| Use multipart upload for large files | Parallel parts, cheap retries |
| Read large objects with parallel range requests | Saturate the network instead of one connection |
| Spread key prefixes under heavy load | Use more index partitions at once |
| Send and verify checksums | End-to-end integrity, not just “upload succeeded” |
| Enable versioning and object lock for critical data | Durability does not protect against your own delete |
| Replicate across regions for disaster recovery | One region is one failure domain |
| Choose storage classes by access pattern | Cold data should cost cold-data prices |
Do
Design keys for distribution, verify checksums on upload and download, protect important data from deletion with versioning, and replicate across regions when a regional outage would be unacceptable.
Don't
Treat eleven nines as protection against mistakes. The system will durably store your accidental overwrite or delete exactly as faithfully as anything else.
Takeaway
An object store is a sharded index in front of an erasure-coded disk fleet, kept honest by constant checksums and automatic repair. That explains why prefix design affects speed, why multipart uploads help, and why durability and availability are separate numbers. Design your keys and your safeguards with that structure in mind.
Frequently asked questions
What does eleven nines of durability mean?
A design target of 99.999999999 percent annual durability for an object: the expected yearly loss is about one object in a hundred billion. It describes durability, meaning data is not lost, and is separate from availability, meaning you can read it right now.
What is erasure coding and why use it instead of copies?
Erasure coding splits data into fragments and adds computed parity fragments so the original can be rebuilt from any sufficient subset. It tolerates several lost disks or zones with far less storage overhead than keeping full copies, at the cost of more computation and network use when rebuilding.
Why do common key prefixes cause hotspots in object storage?
The key index is partitioned by key ranges, so keys that share a prefix and arrive at a high rate concentrate on the same index partition. Spreading keys across many prefixes, with high-cardinality parts early in the name, lets requests land on different partitions and scale in parallel.
Why does multipart upload help?
It splits a large object into parts uploaded in parallel and retried individually, which raises throughput and means a network failure costs one part instead of the whole upload. Parts are assembled into a single object when you complete the upload.

/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.