Moving Live Data Between Shards With Zero Downtime: The Six-Step Protocol
Register, bulk copy, replicate the tail, verify, flip traffic behind a version gate, clean up. How online shard migrations work without losing a write.
On this page
Sharding gets the attention, but the harder skill is moving data between shards while the product keeps taking writes. Splitting a hot shard, packing quiet ones together, upgrading an engine: all of them are the same operation, and getting it wrong means either downtime or a lost payment.
The requirements that shape the design
A migration platform worth trusting meets four constraints at once.
Vocabulary
A chunk is a slice of a collection that lives on one shard. A shard is a replica set: one primary taking writes plus secondaries. Routing (which chunk lives on which shard) is held in a metadata service, and a fleet of proxies reads it to send each query to the right place. Moving a chunk means changing that mapping, safely.
The six steps
- 1. Register the migration, build indexes first
Record the intent to move chunks from source to target, then build the indexes on the target before any data arrives. Queries on newly arrived data are then fast from the first moment instead of scanning until indexes finish, and the cutover does not change query behaviour.
- 2. Bulk-import a snapshot
Load a point-in-time snapshot of the chunks onto the target, optionally filtered. Engineers hit a throughput wall here. Batching writes and tuning engine parameters did not help. What did: sort the data by the most common index attributes and insert in that order. B-tree writes land on neighbouring pages, and Stripe reported about a tenfold write-throughput improvement.
- 3. Replicate everything after the snapshot
Writes keep arriving at the source. Each shard already keeps an operation log (the oplog in MongoDB) of every mutation. Ship that log to a message broker such as Kafka, archive it to object storage, and run a replication service that applies it to the target. Because it reads the log rather than the live collections, it consumes none of the source’s read capacity. Replication runs in both directions so traffic can be reversed later.
- 4. Verify correctness
Compare point-in-time snapshots of source and target to prove completeness and equality. Doing it from snapshots, not live scans, was a deliberate choice to protect shard throughput.
- 5. Switch traffic behind a version gate
Brief pause, reroute, resume. Detailed below.
- 6. Deregister and clean up
Mark the migration complete in the metadata service and drop the chunk data from the source to reclaim space.
The traffic switch is the whole trick
The dangerous moment is cutover. If one proxy still believes the chunk lives on the source while another writes to the target, you have split-brain and a diverged chunk.
The fix is a version gate. Every proxy attaches a version token to each request. The shard checks the token and serves the request only if it is current.
1. Raise the version token on the SOURCE shard
-> all reads and writes to the chunk there are now rejected
2. Wait for replication to drain the last outstanding writes
3. Update the route in the metadata service to point to the TARGET,
and bump the token there
4. Proxies fetch the new route and token, and send traffic to the TARGET
Requests that fail during the gap are retried, and the retries land on the target and succeed. The protocol described publicly takes under two seconds. The key property: after step 1 it is impossible for any client to write to the old location, regardless of what its cached routing says.
Routing caches lag. Without a check at the shard itself, a slow proxy keeps writing to the source after the move, and those writes are silently lost when the source chunk is dropped. The shard, not the client, must be the authority on whether a request is current.
What the same machinery buys you
Once moves are cheap and safe, they become an everyday tool.
| Use | What it does |
|---|---|
| Split a hot shard | Move half its chunks to a new shard during a traffic surge |
| Bin packing | Consolidate lightly used databases onto fewer shards when traffic is low, saving cost |
| Isolate a large tenant | Move a customer from a shared shard to a dedicated one |
| Engine upgrades | Migrate to a freshly upgraded fleet instead of upgrading in place |
Apply it without a platform team
You do not need thousands of shards to use the pattern. Any live data move, such as changing databases or splitting a service’s tables out, follows the same shape.
Do
Snapshot, then tail the change log from the snapshot point. Verify before you cut over. Gate writes at the destination, not just in the client. Keep reverse replication for a defined soak period before deleting the source.
Don't
Copy the table, flip a config flag, and hope nothing wrote in between. The writes that landed during the copy are precisely the ones you lose.
Takeaway
Online migration is a protocol, not a script: an indexed target, a fast ordered bulk load, a replayed change log, a verification pass, a version-gated flip, and a cleanup. Build it once as a platform capability and rebalancing stops being an event, which is what lets a sharded system keep growing without ever scheduling downtime.
Frequently asked questions
How do you migrate data between shards without downtime?
Copy a snapshot in bulk, then stream every change made after the snapshot from the source's operation log until the target has caught up. Verify both sides match, briefly block writes on the source, finish replication, and switch routing. The write pause lasts seconds, and clients retry through it.
What is a version gate in a traffic switch?
A version number that each request carries and the shard checks before serving it. Raising the number on the source makes it reject requests routed with stale routing information, so no client can write to the old location after the cutover begins.
Why copy data in sorted order when bulk loading?
Databases keep data in B-tree structures. Inserting in key order writes to neighbouring pages, which improves cache use and avoids page splits. Stripe reported roughly a tenfold gain in write throughput from sorting by the most common index attributes before inserting.
Why replicate in both directions after cutover?
It keeps a rollback path open. If the target shard misbehaves after traffic moves, you can route traffic back to the source, which still holds all the writes made during the trial period, instead of losing them or restoring from backup.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.