---
answer: direct
beat: system-design
source: 1 article · updated: November 14, 2026
---

Why load only delta data into a store instead of the full dataset?

Most keys do not change between loads. Publishing only changes cuts transfer and load time, reduces pressure on the serving nodes and lowers cost. One reported change reduced a daily load from about 4 TB to 40 to 80 GB.

Answered in

Serving Derived Data at Scale: From a Read-Only Store to a Unified Key-Value Engine

Three generations of a derived-data store: read-only files, a batch plus real-time merge, then hash shards, a Kafka log and delta bulk loads.

Crashtech Editorial November 14, 2026 System Design & Architecture

Read the full analysis

Other questions this article answers

More system design & architecture questions

Every answer on Crashtech is written by the editor of the article it comes from — never auto-summarised. Browse all answers or the System Design & Architecture beat.