Streaming vs Batch: Managing the Trade-Off Between Latency and Completeness
Why data architectures split between streaming and batch: master tumbling windows, watermarks, late data arrival, and exactly-once processing.
On this page
Every data platform architecture is fundamentally governed by a single engineering compromise: The Trade-Off Between Latency and Completeness.
If you need a calculation right now (sub-second latency), you must accept incomplete data: network packets from mobile devices are still in transit, transactions are queued in message brokers, and clock skew means events arrive out of order.
Conversely, if you require 100% mathematical completeness (such as settling end-of-month bank ledgers or training recommendation models), you must wait hours for late-arriving records to settle before executing a batch job.
Understanding the architectural mechanics of Streaming (Apache Flink, Spark Streaming) vs Batch (Snowflake, BigQuery, dbt) is essential to building reliable data pipelines.
The Spectrum of Data Processing
+-----------------------------------------------------------------------+
| Continuous Real-Time Streaming (Flink / Kafka Streams) |
| - Ingestion: Record-by-record event stream |
| - Latency: 5ms to 200ms |
| - Completeness: Speculative (Requires watermarks for late data) |
| - Use Case: Fraud detection, algorithmic trading, live alerts |
+-----------------------------------------------------------------------+
| Micro-Batching (Spark Streaming) |
| - Ingestion: Accumulates small time slices (e.g. 500ms to 5s) |
| - Latency: 1 second to 10 seconds |
| - Use Case: Near real-time dashboards, operational metric feeds |
+-----------------------------------------------------------------------+
| Traditional Batch Processing (Snowflake / Databricks) |
| - Ingestion: Periodic scheduled sweeps (Hourly / Daily / Monthly) |
| - Latency: 1 hour to 24 hours |
| - Completeness: 100% deterministic (All late data captured) |
| - Use Case: Financial accounting, regulatory audits, deep ML training|
+-----------------------------------------------------------------------+
The Real-Time Dilemma: Windows, Watermarks, and Late Arrival
In a pure batch architecture, time is simple: you run a query over records where date = '2026-07-08'.
In a streaming pipeline, data arrives continuously over an unbounded stream. To calculate an aggregate (such as average transactions per minute), the engine must partition continuous time into discrete Windows:
1. Tumbling Window: Non-overlapping fixed blocks (e.g., [12:00 - 12:05], [12:05 - 12:10])
2. Sliding Window: Overlapping intervals (e.g., last 5 minutes, recalculated every 10 seconds)
3. Session Window: Dynamic gaps triggered by periods of user inactivity
Event Time vs Processing Time
The hardest challenge in stream processing is the discrepancy between two timestamps:
- Event Time: The exact millisecond the user clicked the button on their phone (
12:01:05.100). - Processing Time: The millisecond the event reaches the Flink worker in AWS (
12:01:28.400because the phone entered a subway tunnel).
If your streaming window closes at 12:05:00 based on Processing Time, the subway user’s events are either omitted or corrupt the next window.
Watermarks: The Clock of the Unbounded Stream
To solve out-of-order data, stream engines use Watermarks.
Watermark(t) = A signal asserting that the engine expects no more events with EventTime < t
If a system sets a 10-second watermark delay:
- At
12:05:10, the engine officially closes the[12:00 - 12:05]window and emits the aggregate. - Any record arriving after the watermark is classified as Late Data and routed to a Dead Letter Queue (DLQ) or triggers an incremental window retraction.
The Modern Convergence: The Kappa Architecture
Historically, companies operated a fragile Lambda Architecture: running a real-time streaming layer for fast estimates and a separate batch layer to overwrite estimates overnight with accurate calculations. Maintaining two independent codebases for the same business metric inevitably led to synchronization bugs.
Today, data engineering standardizes on the Kappa Architecture:
+-----------------------------------------------------------------------+
| The Kappa Architecture |
+-----------------------------------------------------------------------+
| |
| [ Inbound Events Stream ] |
| | |
| v |
| [ Immutable Distributed Event Log (Kafka / Redpanda) ] |
| | (Stores raw events with long retention) |
| | |
| +---> Real-Time Flink Job -> Low-Latency Real-Time Views |
| | |
| +---> Open Table Storage (Apache Iceberg / Delta Lake) |
| | |
| v |
| Ad-hoc SQL Analytics (Snowflake / Trino / DuckDB) |
+-----------------------------------------------------------------------+
By storing immutable events in an append-only log and utilizing modern open table formats (Apache Iceberg), the same underlying data serves both sub-second streaming consumers and petabyte-scale historical batch queries.
Key Engineering Takeaways
- Latency and completeness are opposites: You cannot have instantaneous real-time metrics that are guaranteed 100% complete. Design your business rules around watermarks.
- Always differentiate Event Time from Processing Time: Cell phones lose signal and network switches queue packets. Always compute business analytics against Event Time.
- Ditch Lambda for Kappa: Avoid maintaining duplicate streaming and batch codebases. Use immutable event logs (Kafka) paired with open table formats (Iceberg) to unify data pipelines.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.