Serving Derived Data at Scale: From a Read-Only Store to a Unified Key-Value Engine
Three generations of a derived-data store: read-only files, a batch plus real-time merge, then hash shards, a Kafka log and delta bulk loads.
On this page
Not all data in production is entered by users. A great deal is computed overnight or on a stream and then looked up on every page view. That derived data has its own requirements, and generic databases rarely fit them. The story of how one company built three successive stores for it is a compact lesson in system evolution.
What the store must do
| Requirement | Why |
|---|---|
| Petabyte scale | Derived datasets are large and growing |
| Fast bulk upload | Offline jobs produce huge outputs on a schedule |
| Low-latency reads | They sit on the request path of user-facing pages |
| High availability | Missing recommendations degrade the product visibly |
| Multi-tenancy | Many teams share one platform |
Existing options (a relational database, HBase, an embedded engine) could not meet all of them at once, which is how a custom store happens.
Generation 1: read-only files
The first service served immutable files in a format derived from Google’s SSTable, built by batch jobs and uploaded to object storage daily. Servers downloaded their assigned shards and swapped out the old data.
- Sharding. A fixed number of shards by primary key, each replicated across servers.
- Partition mapping. A coordination service held the shard-to-server map, updated by hand.
- Batch updates. A daily Hadoop job produced files for the next day.
It scaled reads well, but data was a day stale and could not be updated at all. Anything real-time was impossible.
Generation 2: batch plus real time, merged on read
The second system added a fast NoSQL database for real-time writes beside the batch file store, and merged results by timestamp at query time. A daily Spark job pre-merged both into a single dataset to reduce the merge work per query.
Generation 3: one store for everything
A unified design, introduced in 2018, answers each earlier weakness.
- Automatic partitioning: 1,024 hash shards
Raise shards from 8 to 1,024 and partition by hash of the primary key. A cluster manager (Apache Helix) decides which node holds which shard and rebalances automatically, replacing the hand-edited map.
- Writes go through a Kafka log
Every write is first appended to a log with 1,024 partitions, aligned one to one with shards. Each shard’s replicas read their partition in order, so all replicas apply the same sequence of changes. This absorbs write spikes.
- Leaderless replication, eventual consistency
Any replica of a shard can serve reads. There is no per-shard leader to fail over, which favours availability for read-heavy workloads. Derived data tolerates short staleness because it can be recomputed.
- One LSM storage engine for batch and real time
The real-time database is gone. Instead, a storage engine from HBase (an LSM tree with an in-memory write buffer and a block cache) serves everything. Each client table maps to a column family, supporting point lookups, key-range scans and time-range queries.
- Compaction without hurting reads
LSM trees accumulate small files and expired data, which must be merged (compaction). Nodes are split into online nodes, which serve reads with throttled compaction, and batch nodes, which compact at full speed. The cluster manager rotates roles on a schedule.
- Delta bulk loads
Two pipelines load warehouse output: merge (combine with existing data by timestamp) and replace (import and drop all older versions). Spark prepares files in the engine’s format, uploads them to object storage, and each node bulk-loads them. Loading only the changed keys cut a daily load from around 4 TB to 40 to 80 GB.
Eventual consistency here is a feature, not a flaw. Derived data is rebuildable, read-heavy and tolerant of seconds of staleness, so giving up leader-based strong consistency buys availability and simplicity. A payments ledger would choose differently.
The numbers, and what is still missing
| Metric | Reported value |
|---|---|
| Data stored | About 130 TB across 4,000 tables |
| Availability | Above 99.9 percent |
| Read throughput | Over 800,000 queries per second |
| Write throughput | About 35,000 writes per second |
| p95 read latency | Under 8 ms |
On the roadmap: read-after-write consistency, autoscaling and repartitioning driven by traffic instead of fixed shard counts.
Patterns you can reuse
| Pattern | Where it showed up |
|---|---|
| Over-partition early | 1,024 logical shards on far fewer machines, so rebalancing moves shards, not data layout |
| One log, one partition per shard | Ordering per shard and a natural replication source |
| Separate bulk and online work | Different node roles for compaction and for serving |
| Publish deltas, not snapshots | Cost scales with change, not with total size |
| Retire a system by absorbing it | Removing the second database lowered operations and merge latency |
Do
Write down your consistency needs per dataset before choosing replication, size the shard count for years of growth, automate placement from day one, and measure how much of each load is actually new.
Don't
Run two stores and merge on read because it avoids a migration today. The merge latency and the operational double bill will arrive on every request and every month.
Takeaway
Each generation of this store fixed what the last one exposed: freshness, then operational complexity, then wasted bulk work. The end state is a small set of strong ideas: hash shards with automatic placement, an ordered log feeding leaderless replicas, an LSM engine with roles split between serving and compaction, and loads that move only what changed.
Frequently asked questions
What is derived data?
Data computed from larger raw datasets, either by offline batch jobs or by processing event streams, and then served to applications. Examples are per-user recommendations, personalisation features and aggregates. It is rebuildable from source, so it can trade strict consistency for availability and speed.
Why use Kafka as a write-ahead log for a key-value store?
Writes are first appended to a durable ordered log, one partition per shard, and each replica applies them in order. That smooths write spikes, gives every replica the same sequence for eventual convergence, and lets any replica serve reads without a leader.
Why load only delta data into a store instead of the full dataset?
Most keys do not change between loads. Publishing only changes cuts transfer and load time, reduces pressure on the serving nodes and lowers cost. One reported change reduced a daily load from about 4 TB to 40 to 80 GB.
Why separate online and batch nodes?
LSM-based storage needs compaction, which competes with reads for CPU and disk. Dedicating some nodes to fast compaction while others serve reads, and rotating them, keeps read latency steady and still lets files merge at full speed.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.