---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 4 min
---

How Object Storage Reaches Eleven Nines: Index, Erasure Coding and Self-Healing

What happens when you PUT an object: routing, authorisation, a sharded metadata index, erasure-coded placement across zones, and background auditors.

– –

The first object store launched with a promise: put any amount of data in, get it back later. Nearly two decades on, it holds hundreds of trillions of objects. Knowing the machinery behind a simple PUT explains most of its quirks, and most of the performance advice.

What a write actually does

  1. The client sends a PUT

    Through an SDK, CLI or REST API. Large objects use multipart upload: split into parts, upload in parallel, retry parts independently, then complete the upload so parts become one object.

  2. DNS and routing

    The bucket’s hostname resolves to a nearby entry point. Features such as multi-region access points can route to the lowest-latency region.

  3. Front-end servers check the request

    Identity and permissions are verified (IAM, bucket policy), headers and signatures validated, and the request goes to a front-end server chosen by load balancing. An unauthorised request ends here with a 403.

  4. The key is recorded in the index

    A metadata store maps the key to where its bytes live, along with size, version and permissions.

  5. Data is placed with erasure coding

    The object is split into fragments with computed parity, spread over many disks and several availability zones.

  6. The client gets a confirmation

    Including an ETag (a checksum) and, with versioning enabled, a VersionId. Reads after this see the new object: modern S3 provides strong read-after-write consistency.

The index: why prefixes matter

Looking up a key must not scan billions of entries. The metadata index is partitioned by key range (lexicographic order), so each partition handles a slice of the keyspace. Two things follow.

  • Requests spread across many partitions run in parallel, which is how request rates reach very high levels.
  • If most of your traffic shares one prefix, such as every key starting with the same date, those requests pile onto one partition. The service rebalances partitions as load grows, but a sudden, concentrated burst can still hit throttling first.
The practical fix

Put high-cardinality characters early in the key (a hash or short random prefix, a customer id) and spread load across many prefixes. Remember per-prefix request limits scale with the number of prefixes you use, so parallel prefixes mean parallel capacity.

hot:   logs/2026-11-14/00001.json  logs/2026-11-14/00002.json   (one prefix, one partition)
spread: 3f/logs/2026-11-14/00001.json  a9/logs/2026-11-14/00002.json

Erasure coding in one picture

Keeping three full copies tolerates two failures at 200 percent overhead. Erasure coding gets similar or better resilience for less storage.

data split into k fragments + m parity fragments = k + m pieces
any k of the k + m pieces can rebuild the object
example: k = 6, m = 3  ->  survive any 3 lost pieces, overhead 50%
Full replication (3 copies)Erasure coding (6+3)
Failures tolerated23
Storage overhead200%50%
Read costRead one copyRead k pieces, reassemble
Repair costCopy a surviving replicaRead k pieces to rebuild one

The price is compute and network work on rebuild. Placing pieces across zones means losing a whole zone is just losing a few pieces.

Advertisement

Where durability really comes from

Eleven nines is not one clever trick. It is redundancy plus vigilance.

Redundancy across failure domains disks, racks, zones
Fragments are spread so no single disk, rack or zone failure can take out enough pieces to lose data.
Continuous integrity checks checksums
Stored data is repeatedly verified against checksums, and the checksum you can request on upload proves end-to-end integrity from your client to disk.
Background auditors and repair self-healing
Scanners detect corrupted or missing fragments and trigger automatic rebuilds long before a second failure could cause loss.
Fast repair beats rare failure the math
Durability depends on repairing faster than failures accumulate. Quicker detection and rebuild lowers the odds that enough pieces fail together.

Durability and availability are different promises. An object can be perfectly safe and temporarily unreachable.

Handling a hundred million requests a second

No single machine does this. The system relies on:

  • Parallel index partitions, each serving a range of keys.
  • Caching for hot metadata and data paths.
  • Dynamic rebalancing, splitting partitions that run hot.
  • Latency-aware routing to steer requests to healthy, fast paths.
  • Many small services, each scaled and deployed independently, so one component’s trouble stays local.

What to do as a user

PracticeWhy
Use multipart upload for large filesParallel parts, cheap retries
Read large objects with parallel range requestsSaturate the network instead of one connection
Spread key prefixes under heavy loadUse more index partitions at once
Send and verify checksumsEnd-to-end integrity, not just “upload succeeded”
Enable versioning and object lock for critical dataDurability does not protect against your own delete
Replicate across regions for disaster recoveryOne region is one failure domain
Choose storage classes by access patternCold data should cost cold-data prices

Do

Design keys for distribution, verify checksums on upload and download, protect important data from deletion with versioning, and replicate across regions when a regional outage would be unacceptable.

Don't

Treat eleven nines as protection against mistakes. The system will durably store your accidental overwrite or delete exactly as faithfully as anything else.

Takeaway

An object store is a sharded index in front of an erasure-coded disk fleet, kept honest by constant checksums and automatic repair. That explains why prefix design affects speed, why multipart uploads help, and why durability and availability are separate numbers. Design your keys and your safeguards with that structure in mind.

Advertisement

Frequently asked questions

What does eleven nines of durability mean?

A design target of 99.999999999 percent annual durability for an object: the expected yearly loss is about one object in a hundred billion. It describes durability, meaning data is not lost, and is separate from availability, meaning you can read it right now.

What is erasure coding and why use it instead of copies?

Erasure coding splits data into fragments and adds computed parity fragments so the original can be rebuilt from any sufficient subset. It tolerates several lost disks or zones with far less storage overhead than keeping full copies, at the cost of more computation and network use when rebuilding.

Why do common key prefixes cause hotspots in object storage?

The key index is partitioned by key ranges, so keys that share a prefix and arrive at a high rate concentrate on the same index partition. Spreading keys across many prefixes, with high-cardinality parts early in the name, lets requests land on different partitions and scale in parallel.

Why does multipart upload help?

It splits a large object into parts uploaded in parallel and retried individually, which raises throughput and means a network failure costs one part instead of the whole upload. Parts are assembled into a single object when you complete the upload.

Sources & further reading

/* Comments */