---
topic: system-design
author: Crashtech Editorial
published: Oct 29, 2026 · read: 3 min
---

Data Movement Strategies in System Design: Pull, Poll, Push, and Event Streams

A comprehensive guide to moving data across distributed systems: comparing synchronous REST polling against WebSockets, SSE, Kafka, and Change Data Capture.

– –

At the heart of every software architecture lies a single operational mandate: moving data from where it is created to where it is needed.

A user clicks a button on a web browser, a payment transaction commits in PostgreSQL, a driver’s GPS coordinates change on an iPhone, an IoT sensor emits a temperature spike. For the broader system to behave as a coherent, unified entity, that information must travel to downstream services, analytics warehouses, search indexes, and mobile screens.

Choosing how that data moves: whether a consumer pulls it on-demand, polls repeatedly on a timer, waits on a persistent push stream, or ingests an asynchronous event log, determines the latency, scalability, network bandwidth, and failure resilience of your entire infrastructure.

The Taxonomy of Data Movement

Every data movement pattern can be characterized by three architectural components:

Source (Originator) ===[ Transport Protocol / Channel ]===> Consumer (Destination)
                        Trigger: Clock / State Change / Request

The fundamental choice comes down to who initiates the transfer and how the transport connection is managed:

+-----------------------------------------------------------------------------+
|                          DATA MOVEMENT TAXONOMY                             |
+-----------------------------------------------------------------------------+

 Strategy      Initiator    Connection Lifecycle    Latency       Network Overhead
 -----------------------------------------------------------------------------
 Short Poll    Consumer     Closed per request      High (Interval) Massive (Empty requests)
 Long Poll     Consumer     Held until data arrives Medium        High reconnect rate
 Server-Sent   Server       Persistent HTTP pipe    Low (<50ms)   Minimal (One-way push)
 WebSockets    Bidirectional Persistent TCP socket  Ultra-Low     Minimal (Dual-direction)
 CDC Streaming System       Persistent binary log   Near-Realtime Zero polling load on DB

Strategy 1: The Trap of Short Polling vs Long Polling

In early web architectures, if a frontend needed to know when a background job completed, the client ran an HTTP short-polling loop:

setInterval(() => {
  fetch('/api/job/8812/status'); // Fires every 2,000ms
}, 2000);

The Cost of Short Polling

If the background job takes 40 seconds to process:

  • The client fires 20 separate HTTP requests.
  • 19 of those requests return {"status": "PENDING"}.
  • 95% of server CPU cycles, TLS handshakes, and database queries are completely wasted processing empty negative responses.

Long Polling: The Intermediate Improvement

With HTTP Long Polling, the client issues an HTTP request, but the server holds the request open until state changes or a 30-second timeout occurs. The moment the background job finishes, the server flushes the response and closes the connection. While long polling eliminates empty responses, it introduces significant connection churn as clients constantly re-establish TLS handshakes after every message.

Strategy 2: Server-Sent Events (SSE) vs WebSockets

When building modern interactive user interfaces (live financial tickers, AI chat streaming, sports scoreboards), persistent push channels replace polling entirely:

+-----------------------------------------------------------------------------+
|                         SSE VS WEBSOCKETS IN PRODUCTION                     |
+-----------------------------------------------------------------------------+

 Dimension             Server-Sent Events (SSE)       WebSockets
 -----------------------------------------------------------------------------
 Protocol              Standard HTTP/2 or HTTP/1.1    Custom WebSocket Protocol
 Directionality        Unidirectional (Server -> Client) Full-Duplex (Bidirectional)
 Firewall Traversal    Zero issues (Standard Port 443) Can be blocked by enterprise proxies
 Reconnection          Automatic (Native in browsers) Requires custom client logic
 Payload Format        UTF-8 Text Streams             Binary (Protobuf) or Text
 Best Suited For       LLM Token Streaming, Feeds     Live Chat, Multiplayer Games

For 80% of web push applications: including generative AI token streaming, Server-Sent Events (SSE) is the superior architectural choice. It runs over standard HTTP, traverses corporate firewalls without proxy issues, and browsers natively handle automatic reconnection and message ID tracking out of the box.

Strategy 3: Change Data Capture (CDC) and Event Logs

At the enterprise infrastructure layer, moving data between databases and downstream analytics systems cannot use application-level polling (e.g., querying WHERE updated_at > ?):

  • Polling misses intermediate state updates (if a row changes from A -> B -> C between poll intervals, the consumer only sees C, losing the audit trail of B).
  • Table scans saturate database memory buffers and lock row indices.

Modern data platforms deploy Log-Based Change Data Capture (CDC) (such as Debezium running on Kafka Connect):

 [PostgreSQL Primary Engine]
             |
             | 1. Writes byte-level transaction delta
             v
 [Physical Write-Ahead Log (WAL) on Disk]
             |
             | 2. Non-blocking binary log tailing
             v
 [Debezium CDC Connector]
             |
             | 3. Emits structured JSON/Avro events
             v
 [Apache Kafka Distributed Log]
             |
             +---> [Consumer 1: Real-time Elasticsearch Indexer]
             +---> [Consumer 2: Snowflake Analytics Pipeline]
             +---> [Consumer 3: In-Memory Redis Cache Evictor]

CDC decouples data movement from primary application code:

  1. The application simply writes to PostgreSQL and returns.
  2. The CDC engine tails the physical database Write-Ahead Log asynchronously without placing any lock overhead on the relational engine.
  3. Every single INSERT, UPDATE, and DELETE is transformed into an ordered, immutable event stream on Kafka, allowing downstream data lakes, caches, and search clusters to synchronize in near-real time.

Decision Matrix: Choosing the Right Movement Strategy

Does data move from Server to Browser?
  - Need bidirectional client messages (e.g., game controls)? ===> WebSockets
  - Only server push (e.g., AI token stream, notifications)? ===> Server-Sent Events (SSE)

Does data move between Database and Internal Microservices?
  - Need audit trail, zero DB lock overhead, real-time?   ===> Change Data Capture (CDC)
  - Inter-service asynchronous commands?                  ===> Message Queue (RabbitMQ / SQS)

Does data move across Analytical Data Warehouses?
  - Low-latency real-time stream processing?              ===> Apache Flink / Kafka Streams
  - Massive historical analytics (batch aggregations)?    ===> Micro-batch ELT (Airbyte / dbt)

Architectural Takeaway

Data movement is the circulatory system of distributed architecture:

  • Never use short polling in production; it wastes 90% of network and compute resources.
  • Adopt Server-Sent Events (SSE) as the default for server-to-client streaming before jumping to complex bidirectional WebSockets.
  • Use log-based Change Data Capture (CDC) off the database Write-Ahead Log to keep downstream search indices, caches, and analytics stores synchronized without polling overhead.
Advertisement

/* Comments */