A Real-Time Data Stack, Layer by Layer: Kafka, Flink, Pinot and the Trade-offs Between Them
Messaging, stream processing and real-time OLAP each solve one problem. How to pick consistency or availability per use case, and replay with Kappa+.
On this page
Streaming demos show an event arriving and a chart moving. Real platforms spend most of their design effort on what happens when a region dies, when a bug needs three days of data reprocessed, or when a dashboard has to show a corrected value. The layers are simple. The trade-offs are the architecture.
Five requirements that pull against each other
| Requirement | What it means in practice | Pulls against |
|---|---|---|
| Consistency | No loss, no duplicates across regions, critical for financial data | Availability, freshness |
| Availability | Four nines, no downtime for things like dynamic pricing | Consistency, cost |
| Freshness | Events usable within seconds, queries under a second | Cost, consistency |
| Scalability | Absorb growth and new use cases without redesign | Simplicity |
| Cost efficiency | Memory and storage are the bill | Everything above |
No use case gets all five. The skill is deciding which two or three each one needs most.
The layers
What each use case demands
| Use case | Pipeline | What matters most |
|---|---|---|
| Dynamic pricing | Kafka to Flink, multiplier per area | Freshness and availability over strict consistency |
| Restaurant dashboard | Pinot with pre-aggregated data | Sub-second queries, fresh data |
| ML prediction monitoring | Flink aggregates predictions against actual outcomes | Throughput, millions of points per second |
| Operations automation | Presto over Pinot for ad hoc queries, triggers on thresholds | Flexible querying on live data |
| Payments and audit | Strongly consistent path | No loss, no duplication, order |
Pre-aggregation is the quiet trick behind fast dashboards: compute the metrics you will ask for once, at ingestion, and store them, rather than scanning raw events per request.
Multi-region: choose per workload
Active-active runs processing in several regions at once, with message replication between clusters. If a region fails, another already has the data and keeps going. This suits pricing: it cannot afford downtime, and a briefly different view between regions is acceptable.
Active-passive lets one region consume at a time, and on failure a backup resumes at the exact point the primary stopped. This is for workloads where loss or duplication is unacceptable, like payments. The hard part is offset synchronisation: the consumer in the backup region must start at the position that corresponds to where the primary left off, even though offsets differ between clusters. Replication tools periodically checkpoint offset mappings so failover can translate between them.
Write down, per pipeline, whether it is availability-first or consistency-first. Teams that never decide discover their answer during the first regional outage, and the wrong default is expensive both ways.
One codebase for live and replay: Kappa+
Bugs get fixed, models need retraining, new pipelines need testing on history. Maintaining one code path for streaming and another for batch doubles the work and creates drift between them.
The Kappa idea is to run one stream-processing implementation and replay data through it. Kappa+ extends it to replay from archived data in the lake, not just retained Kafka logs. That needs tuning for batch-like input: bigger buffers, handling of out-of-order records, and throttling so a replay does not swamp downstream systems.
live: Kafka topic --> Flink job (logic X) --> sink
backfill: lake files --> Flink job (logic X, tuned for bounded input) --> sink
The payoff is that a backfill verifies the same logic that runs in production.
Lessons that transfer to any scale
- Adopt open source, expect to customise it
The products are strong foundations, and scale-specific needs such as federation, upsert and SQL layers were built on top.
- Standardise thin clients and invest in CI/CD
A consistent client interface and automated test and deploy let you change internals without breaking dozens of consumers.
- Automate operations
Scaling, cluster management and recovery by hand does not survive growth. Add monitoring and alerting before you need them.
- Automate onboarding and discovery
Self-service topic creation, job launch and data discovery let teams use the platform without queuing at the data team.
Do
Classify each pipeline as consistency-first or availability-first, plan the failover path for each, keep stream and replay logic identical, and treat dead letter queues and checkpoints as first-class parts of the design.
Don't
Build a streaming stack for a dashboard that refreshes hourly. A scheduled query on a warehouse is simpler, cheaper and easier to fix.
Takeaway
Messaging, stream processing and real-time OLAP are three separate answers to three separate questions, with a lake behind them for history. The architecture lives in the decisions between them: availability or consistency per use case, active-active or active-passive per region, one codebase for live and replay. Make those decisions explicitly, and the products are the easy part.
Frequently asked questions
What are the three layers of a real-time data platform?
Messaging (Kafka) moves events durably, stream processing (Flink) transforms and aggregates them continuously, and real-time OLAP (Pinot) serves low-latency analytical queries on fresh data. A long-term store such as object storage or HDFS supports backfill, checkpoints and model training.
When should a pipeline favour availability over consistency?
When a slightly imperfect answer now beats a perfect answer later. Dynamic pricing or demand heatmaps tolerate a lost or duplicated message, but cannot tolerate downtime. Payments and financial reporting need the opposite: no loss, no duplication, even if failover is slower.
What is Kappa architecture and what does Kappa+ add?
Kappa runs one stream-processing codebase over an append-only log, replaying the log to reprocess history, instead of maintaining separate batch and streaming code. Kappa+ extends this to replay from archived data in the lake, using the same logic with tuning for bounded, out-of-order historical input.
Why does an OLAP store need upsert support?
Real-world facts change: a fare is corrected, a delivery changes status. Append-only analytics shows stale or duplicate rows. Upsert lets the store keep only the latest version of each key, so dashboards reflect corrections without waiting for a batch rebuild.

/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.