---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 5 min
---

Storing a Growing History: Split by Type, Tier by Age, Filter Before You Write

How a viewing-history store survived explosive growth: hot and compressed tiers, chunked blobs, per-type clusters, rotation by age, and dropping noise early.

– –

Append-mostly history looks harmless: one more row for every thing the user did. A decade later it is the largest dataset you own, nearly all of it untouched, and every read is paying for it. The shape of the cure applies to audit logs, activity feeds, message history and metrics as much as to video.

What made Cassandra a good first fit

Property of the workloadWhy the store matched
Roughly nine writes for every readLog-structured storage is write-optimised
Availability beats strict consistencyResume points can sync a little late, but must always load
History grows without bound per userWide rows hold many records under one key
Many usersPartition by customer id to spread across nodes

One row per customer, one column per viewing record, partitioned horizontally by customer. A new play adds a column, and pausing or stopping updates it.

Where the simple design hurt

Three read patterns existed: fetch everything, fetch a time range, and paginate. Each degraded as histories grew. Underneath:

  • Too many SSTables. Reads had to consult many files per request.
  • Compaction overhead. Merging files took longer and cost more as data grew.
  • Repair cost. Read repair and full repair got expensive.

A cache in front (compressed history per user, merged on write, reloaded from the database on a miss) took read load off the cluster. It bought time. It did not fix the growth.

Generation two: live and compressed

The key observation: most reads want recent data, and the old tail is rarely needed.

Live tier recent, hot
Uncompressed, optimised for fast reads and updates. Frequent compaction, a short grace period so deleted data is purged quickly, and frequent read repair to keep it consistent.
Compressed tier older, cold
Stored compressed to save space and I/O. Rarely updated, so compaction and repair run less often. Optimised for storage efficiency, not latency.

For users with enormous histories, even the compressed blob became a bottleneck. Chunking splits it: a metadata record tracks version and chunk count, each chunk is written separately, and reads fetch the metadata then pull the chunks in parallel. No single huge value stalls a request.

New pressure: growth in kinds of data

Global expansion and a new UI feature (inline video previews) changed the data itself:

  • Previews produced many short records, about 30 percent of the data growth each quarter.
  • Language preferences were duplicated across records.
  • Clients fetched whole histories then filtered, wasting bandwidth.
  • 99th percentile latency became erratic.

Storing everything identically had stopped working.

Generation three: split, shard, filter

  1. Categorise by data type

    Separate clusters for full title plays (the critical data driving resume and recommendations), video previews (high volume, lower value) and language preferences (stored once, with deltas when they change). Each cluster is tuned for its own access pattern and cannot swamp the others.

  2. Shard by data age

    A recent cluster for the past days or weeks, fast reads and writes. A past cluster for months to years, detailed but tuned for slower access. A historical cluster holding compressed summaries of old activity, with minimal detail.

  3. Filter at write time

    Previews watched for only a few seconds are weak signals. Drop them before they ever reach storage.

  4. Expire by policy

    Time-to-live deletes records automatically when they have outlived their usefulness.

  5. Rotate automatically

    A background process moves records from recent to past when they cross an age threshold, and summarises and compresses them into historical later.

Cheapest byte is the one you never store

Filtering and deduplicating at ingestion beats optimising storage later. Every record dropped is also a record you never replicate, compact, repair, cache or read.

Advertisement

Reading across tiers

  • A request for recent activity goes straight to the recent cluster, low latency.
  • A request for history queries tiers in parallel and stitches results together, so even a full-history fetch is fast.
  • The caching layer is built to mirror the storage tiers, with recent and older data cached differently, and the reported cache hit rate reached about 99 percent.

Applying it to your data

QuestionAction
What fraction of reads touch the last 7 days?If it is most, build a hot tier for that window
Which events are noise?Filter or sample at the edge
Does the same attribute repeat in every row?Store it once with deltas
Which data is critical, which is nice to have?Separate clusters or tables with different SLOs
How old can data get before nobody reads it?Set a TTL and a summarisation step
Are individual values getting large?Chunk them and read in parallel

Engines with time-window compaction or native time-series compression make parts of this cheaper to build: group data by time so whole windows expire together instead of being deleted row by row.

Do

Measure read recency distribution first, tier by it, drop low-value events at ingestion, and make data rotation automatic and observable.

Don't

Keep one table forever and tune around it. Compaction, repair and read costs scale with total size, so every year of history taxes today’s requests.

Takeaway

History is not one dataset, it is several with different value curves. Split it by type, tier it by age, filter noise early, expire what no longer matters, chunk what is too big, and mirror the tiers in the cache. Then growth stops being a performance problem and becomes a storage line item.

Advertisement

Frequently asked questions

Why does a wide row of growing history become a problem?

Each user's data accumulates indefinitely, so fetches get slower, reads touch more files, and compaction and repair work grows with the data. Retrieval cost scales with total history even though most requests want only the recent part. The fix is to separate hot recent data from cold old data.

What is the benefit of tiering data by age?

Recent data is read and written constantly and justifies fast settings, while old data is rarely touched and can be compressed, compacted lazily and read in parallel. Tuning each tier for its access pattern cuts storage cost and keeps latency predictable for the common request.

Why filter or expire data before storing it?

Every record you never write costs nothing to store, replicate, compact or read. Low-value events, such as a preview watched for two seconds, can be dropped at ingestion, and records that lose value over time can be expired automatically with a time to live.

What is chunking for large compressed blobs?

Splitting an oversized value into several smaller pieces stored under a metadata record that lists them. Reads fetch the metadata, then retrieve the chunks in parallel, and writes update only what changed. It avoids huge single reads and writes that stall under load.

Sources & further reading

/* Comments */