---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 5 min
---

Serving Derived Data at Scale: From a Read-Only Store to a Unified Key-Value Engine

Three generations of a derived-data store: read-only files, a batch plus real-time merge, then hash shards, a Kafka log and delta bulk loads.

– –

Not all data in production is entered by users. A great deal is computed overnight or on a stream and then looked up on every page view. That derived data has its own requirements, and generic databases rarely fit them. The story of how one company built three successive stores for it is a compact lesson in system evolution.

What the store must do

RequirementWhy
Petabyte scaleDerived datasets are large and growing
Fast bulk uploadOffline jobs produce huge outputs on a schedule
Low-latency readsThey sit on the request path of user-facing pages
High availabilityMissing recommendations degrade the product visibly
Multi-tenancyMany teams share one platform

Existing options (a relational database, HBase, an embedded engine) could not meet all of them at once, which is how a custom store happens.

Generation 1: read-only files

The first service served immutable files in a format derived from Google’s SSTable, built by batch jobs and uploaded to object storage daily. Servers downloaded their assigned shards and swapped out the old data.

  • Sharding. A fixed number of shards by primary key, each replicated across servers.
  • Partition mapping. A coordination service held the shard-to-server map, updated by hand.
  • Batch updates. A daily Hadoop job produced files for the next day.

It scaled reads well, but data was a day stale and could not be updated at all. Anything real-time was impossible.

Generation 2: batch plus real time, merged on read

The second system added a fast NoSQL database for real-time writes beside the batch file store, and merged results by timestamp at query time. A daily Spark job pre-merged both into a single dataset to reduce the merge work per query.

What it fixed freshness
Queries could see data seconds old, merged with the bulk historical snapshot.
What it cost two systems
Two stores to run and keep in sync, and a merge step on the hot path.
The wasteful daily job data volume
Even if 1 to 2 percent of keys changed, the whole dataset was reprocessed and republished each day.
Scaling pain manual operations
Growing data and traffic meant manual maintenance on both halves.

Generation 3: one store for everything

A unified design, introduced in 2018, answers each earlier weakness.

  1. Automatic partitioning: 1,024 hash shards

    Raise shards from 8 to 1,024 and partition by hash of the primary key. A cluster manager (Apache Helix) decides which node holds which shard and rebalances automatically, replacing the hand-edited map.

  2. Writes go through a Kafka log

    Every write is first appended to a log with 1,024 partitions, aligned one to one with shards. Each shard’s replicas read their partition in order, so all replicas apply the same sequence of changes. This absorbs write spikes.

  3. Leaderless replication, eventual consistency

    Any replica of a shard can serve reads. There is no per-shard leader to fail over, which favours availability for read-heavy workloads. Derived data tolerates short staleness because it can be recomputed.

  4. One LSM storage engine for batch and real time

    The real-time database is gone. Instead, a storage engine from HBase (an LSM tree with an in-memory write buffer and a block cache) serves everything. Each client table maps to a column family, supporting point lookups, key-range scans and time-range queries.

  5. Compaction without hurting reads

    LSM trees accumulate small files and expired data, which must be merged (compaction). Nodes are split into online nodes, which serve reads with throttled compaction, and batch nodes, which compact at full speed. The cluster manager rotates roles on a schedule.

  6. Delta bulk loads

    Two pipelines load warehouse output: merge (combine with existing data by timestamp) and replace (import and drop all older versions). Spark prepares files in the engine’s format, uploads them to object storage, and each node bulk-loads them. Loading only the changed keys cut a daily load from around 4 TB to 40 to 80 GB.

Match consistency to the data's nature

Eventual consistency here is a feature, not a flaw. Derived data is rebuildable, read-heavy and tolerant of seconds of staleness, so giving up leader-based strong consistency buys availability and simplicity. A payments ledger would choose differently.

Advertisement

The numbers, and what is still missing

MetricReported value
Data storedAbout 130 TB across 4,000 tables
AvailabilityAbove 99.9 percent
Read throughputOver 800,000 queries per second
Write throughputAbout 35,000 writes per second
p95 read latencyUnder 8 ms

On the roadmap: read-after-write consistency, autoscaling and repartitioning driven by traffic instead of fixed shard counts.

Patterns you can reuse

PatternWhere it showed up
Over-partition early1,024 logical shards on far fewer machines, so rebalancing moves shards, not data layout
One log, one partition per shardOrdering per shard and a natural replication source
Separate bulk and online workDifferent node roles for compaction and for serving
Publish deltas, not snapshotsCost scales with change, not with total size
Retire a system by absorbing itRemoving the second database lowered operations and merge latency

Do

Write down your consistency needs per dataset before choosing replication, size the shard count for years of growth, automate placement from day one, and measure how much of each load is actually new.

Don't

Run two stores and merge on read because it avoids a migration today. The merge latency and the operational double bill will arrive on every request and every month.

Takeaway

Each generation of this store fixed what the last one exposed: freshness, then operational complexity, then wasted bulk work. The end state is a small set of strong ideas: hash shards with automatic placement, an ordered log feeding leaderless replicas, an LSM engine with roles split between serving and compaction, and loads that move only what changed.

Advertisement

Frequently asked questions

What is derived data?

Data computed from larger raw datasets, either by offline batch jobs or by processing event streams, and then served to applications. Examples are per-user recommendations, personalisation features and aggregates. It is rebuildable from source, so it can trade strict consistency for availability and speed.

Why use Kafka as a write-ahead log for a key-value store?

Writes are first appended to a durable ordered log, one partition per shard, and each replica applies them in order. That smooths write spikes, gives every replica the same sequence for eventual convergence, and lets any replica serve reads without a leader.

Why load only delta data into a store instead of the full dataset?

Most keys do not change between loads. Publishing only changes cuts transfer and load time, reduces pressure on the serving nodes and lowers cost. One reported change reduced a daily load from about 4 TB to 40 to 80 GB.

Why separate online and batch nodes?

LSM-based storage needs compaction, which competes with reads for CPU and disk. Dedicating some nodes to fast compaction while others serve reads, and rotating them, keeps read latency steady and still lets files merge at full speed.

Sources & further reading

/* Comments */