---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 5 min
---

A Real-Time Data Stack, Layer by Layer: Kafka, Flink, Pinot and the Trade-offs Between Them

Messaging, stream processing and real-time OLAP each solve one problem. How to pick consistency or availability per use case, and replay with Kappa+.

– –

Streaming demos show an event arriving and a chart moving. Real platforms spend most of their design effort on what happens when a region dies, when a bug needs three days of data reprocessed, or when a dashboard has to show a corrected value. The layers are simple. The trade-offs are the architecture.

Five requirements that pull against each other

RequirementWhat it means in practicePulls against
ConsistencyNo loss, no duplicates across regions, critical for financial dataAvailability, freshness
AvailabilityFour nines, no downtime for things like dynamic pricingConsistency, cost
FreshnessEvents usable within seconds, queries under a secondCost, consistency
ScalabilityAbsorb growth and new use cases without redesignSimplicity
Cost efficiencyMemory and storage are the billEverything above

No use case gets all five. The skill is deciding which two or three each one needs most.

The layers

Messaging: Kafka move events durably
The backbone: trillions of messages a day from apps and services, fanned out to real-time and batch consumers. Large deployments federate into several smaller clusters (around 150 nodes each) for easier scale and operations, add dead letter queues so bad messages do not block traffic, and use a consumer proxy so clients in many languages do not each implement tricky Kafka logic.
Stream processing: Flink compute continuously
Stateful stream jobs with windows and joins. A SQL layer lets non-specialists define jobs. Platform tooling estimates resources per job, auto-scales through the day and restarts failed jobs, which is what makes thousands of jobs operable.
Real-time OLAP: Pinot answer fast
Columnar storage for low-latency analytical queries over fresh data. Upsert support handles corrections. Pair it with a distributed SQL engine for joins it does not support natively, pushing as much work as possible down into Pinot.
Long-term store: HDFS or object storage history and recovery
Kafka data lands as columnar files for backfill, model training and testing pipelines. It also holds checkpoints, the state snapshots from which jobs recover.

What each use case demands

Use casePipelineWhat matters most
Dynamic pricingKafka to Flink, multiplier per areaFreshness and availability over strict consistency
Restaurant dashboardPinot with pre-aggregated dataSub-second queries, fresh data
ML prediction monitoringFlink aggregates predictions against actual outcomesThroughput, millions of points per second
Operations automationPresto over Pinot for ad hoc queries, triggers on thresholdsFlexible querying on live data
Payments and auditStrongly consistent pathNo loss, no duplication, order

Pre-aggregation is the quiet trick behind fast dashboards: compute the metrics you will ask for once, at ingestion, and store them, rather than scanning raw events per request.

Advertisement

Multi-region: choose per workload

Active-active runs processing in several regions at once, with message replication between clusters. If a region fails, another already has the data and keeps going. This suits pricing: it cannot afford downtime, and a briefly different view between regions is acceptable.

Active-passive lets one region consume at a time, and on failure a backup resumes at the exact point the primary stopped. This is for workloads where loss or duplication is unacceptable, like payments. The hard part is offset synchronisation: the consumer in the backup region must start at the position that corresponds to where the primary left off, even though offsets differ between clusters. Replication tools periodically checkpoint offset mappings so failover can translate between them.

Say which side you are on

Write down, per pipeline, whether it is availability-first or consistency-first. Teams that never decide discover their answer during the first regional outage, and the wrong default is expensive both ways.

One codebase for live and replay: Kappa+

Bugs get fixed, models need retraining, new pipelines need testing on history. Maintaining one code path for streaming and another for batch doubles the work and creates drift between them.

The Kappa idea is to run one stream-processing implementation and replay data through it. Kappa+ extends it to replay from archived data in the lake, not just retained Kafka logs. That needs tuning for batch-like input: bigger buffers, handling of out-of-order records, and throttling so a replay does not swamp downstream systems.

live:    Kafka topic  --> Flink job (logic X) --> sink
backfill: lake files  --> Flink job (logic X, tuned for bounded input) --> sink

The payoff is that a backfill verifies the same logic that runs in production.

Lessons that transfer to any scale

  1. Adopt open source, expect to customise it

    The products are strong foundations, and scale-specific needs such as federation, upsert and SQL layers were built on top.

  2. Standardise thin clients and invest in CI/CD

    A consistent client interface and automated test and deploy let you change internals without breaking dozens of consumers.

  3. Automate operations

    Scaling, cluster management and recovery by hand does not survive growth. Add monitoring and alerting before you need them.

  4. Automate onboarding and discovery

    Self-service topic creation, job launch and data discovery let teams use the platform without queuing at the data team.

Do

Classify each pipeline as consistency-first or availability-first, plan the failover path for each, keep stream and replay logic identical, and treat dead letter queues and checkpoints as first-class parts of the design.

Don't

Build a streaming stack for a dashboard that refreshes hourly. A scheduled query on a warehouse is simpler, cheaper and easier to fix.

Takeaway

Messaging, stream processing and real-time OLAP are three separate answers to three separate questions, with a lake behind them for history. The architecture lives in the decisions between them: availability or consistency per use case, active-active or active-passive per region, one codebase for live and replay. Make those decisions explicitly, and the products are the easy part.

Advertisement

Frequently asked questions

What are the three layers of a real-time data platform?

Messaging (Kafka) moves events durably, stream processing (Flink) transforms and aggregates them continuously, and real-time OLAP (Pinot) serves low-latency analytical queries on fresh data. A long-term store such as object storage or HDFS supports backfill, checkpoints and model training.

When should a pipeline favour availability over consistency?

When a slightly imperfect answer now beats a perfect answer later. Dynamic pricing or demand heatmaps tolerate a lost or duplicated message, but cannot tolerate downtime. Payments and financial reporting need the opposite: no loss, no duplication, even if failover is slower.

What is Kappa architecture and what does Kappa+ add?

Kappa runs one stream-processing codebase over an append-only log, replaying the log to reprocess history, instead of maintaining separate batch and streaming code. Kappa+ extends this to replay from archived data in the lake, using the same logic with tuning for bounded, out-of-order historical input.

Why does an OLAP store need upsert support?

Real-world facts change: a fare is corrected, a delivery changes status. Append-only analytics shows stale or duplicate rows. Upsert lets the store keep only the latest version of each key, so dashboards reflect corrections without waiting for a batch rebuild.

Sources & further reading

/* Comments */