---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 5 min
---

Moving Live Data Between Shards With Zero Downtime: The Six-Step Protocol

Register, bulk copy, replicate the tail, verify, flip traffic behind a version gate, clean up. How online shard migrations work without losing a write.

– –

Sharding gets the attention, but the harder skill is moving data between shards while the product keeps taking writes. Splitting a hot shard, packing quiet ones together, upgrading an engine: all of them are the same operation, and getting it wrong means either downtime or a lost payment.

The requirements that shape the design

A migration platform worth trusting meets four constraints at once.

Correct no lost or duplicated writes
Source and target must end up identical. A migration that is fast and 99.99 percent right is a data-loss incident.
Invisible clients never notice
Callers may see a retry. They must never see an outage or a changed answer.
Fast and parallel many chunks at once
Rebalancing thousands of chunks is only practical if each move is cheap and many can run together.
Gentle on the source no extra load
The source shard is still serving production. Migration must not steal its read capacity.

Vocabulary

A chunk is a slice of a collection that lives on one shard. A shard is a replica set: one primary taking writes plus secondaries. Routing (which chunk lives on which shard) is held in a metadata service, and a fleet of proxies reads it to send each query to the right place. Moving a chunk means changing that mapping, safely.

The six steps

  1. 1. Register the migration, build indexes first

    Record the intent to move chunks from source to target, then build the indexes on the target before any data arrives. Queries on newly arrived data are then fast from the first moment instead of scanning until indexes finish, and the cutover does not change query behaviour.

  2. 2. Bulk-import a snapshot

    Load a point-in-time snapshot of the chunks onto the target, optionally filtered. Engineers hit a throughput wall here. Batching writes and tuning engine parameters did not help. What did: sort the data by the most common index attributes and insert in that order. B-tree writes land on neighbouring pages, and Stripe reported about a tenfold write-throughput improvement.

  3. 3. Replicate everything after the snapshot

    Writes keep arriving at the source. Each shard already keeps an operation log (the oplog in MongoDB) of every mutation. Ship that log to a message broker such as Kafka, archive it to object storage, and run a replication service that applies it to the target. Because it reads the log rather than the live collections, it consumes none of the source’s read capacity. Replication runs in both directions so traffic can be reversed later.

  4. 4. Verify correctness

    Compare point-in-time snapshots of source and target to prove completeness and equality. Doing it from snapshots, not live scans, was a deliberate choice to protect shard throughput.

  5. 5. Switch traffic behind a version gate

    Brief pause, reroute, resume. Detailed below.

  6. 6. Deregister and clean up

    Mark the migration complete in the metadata service and drop the chunk data from the source to reclaim space.

The traffic switch is the whole trick

The dangerous moment is cutover. If one proxy still believes the chunk lives on the source while another writes to the target, you have split-brain and a diverged chunk.

The fix is a version gate. Every proxy attaches a version token to each request. The shard checks the token and serves the request only if it is current.

1. Raise the version token on the SOURCE shard
      -> all reads and writes to the chunk there are now rejected
2. Wait for replication to drain the last outstanding writes
3. Update the route in the metadata service to point to the TARGET,
   and bump the token there
4. Proxies fetch the new route and token, and send traffic to the TARGET

Requests that fail during the gap are retried, and the retries land on the target and succeed. The protocol described publicly takes under two seconds. The key property: after step 1 it is impossible for any client to write to the old location, regardless of what its cached routing says.

Why gating beats just updating the route

Routing caches lag. Without a check at the shard itself, a slow proxy keeps writing to the source after the move, and those writes are silently lost when the source chunk is dropped. The shard, not the client, must be the authority on whether a request is current.

Advertisement

What the same machinery buys you

Once moves are cheap and safe, they become an everyday tool.

UseWhat it does
Split a hot shardMove half its chunks to a new shard during a traffic surge
Bin packingConsolidate lightly used databases onto fewer shards when traffic is low, saving cost
Isolate a large tenantMove a customer from a shared shard to a dedicated one
Engine upgradesMigrate to a freshly upgraded fleet instead of upgrading in place

Apply it without a platform team

You do not need thousands of shards to use the pattern. Any live data move, such as changing databases or splitting a service’s tables out, follows the same shape.

Do

Snapshot, then tail the change log from the snapshot point. Verify before you cut over. Gate writes at the destination, not just in the client. Keep reverse replication for a defined soak period before deleting the source.

Don't

Copy the table, flip a config flag, and hope nothing wrote in between. The writes that landed during the copy are precisely the ones you lose.

Takeaway

Online migration is a protocol, not a script: an indexed target, a fast ordered bulk load, a replayed change log, a verification pass, a version-gated flip, and a cleanup. Build it once as a platform capability and rebalancing stops being an event, which is what lets a sharded system keep growing without ever scheduling downtime.

Advertisement

Frequently asked questions

How do you migrate data between shards without downtime?

Copy a snapshot in bulk, then stream every change made after the snapshot from the source's operation log until the target has caught up. Verify both sides match, briefly block writes on the source, finish replication, and switch routing. The write pause lasts seconds, and clients retry through it.

What is a version gate in a traffic switch?

A version number that each request carries and the shard checks before serving it. Raising the number on the source makes it reject requests routed with stale routing information, so no client can write to the old location after the cutover begins.

Why copy data in sorted order when bulk loading?

Databases keep data in B-tree structures. Inserting in key order writes to neighbouring pages, which improves cache use and avoids page splits. Stripe reported roughly a tenfold gain in write throughput from sorting by the most common index attributes before inserting.

Why replicate in both directions after cutover?

It keeps a rollback path open. If the target shard misbehaves after traffic moves, you can route traffic back to the source, which still holds all the writes made during the trial period, instead of losing them or restoring from backup.

Sources & further reading

/* Comments */