---
topic: data-engineering
author: Crashtech Editorial
published: Nov 15, 2026 · read: 4 min
---

Inside Netflix’s Real-Time Graph: Kafka-to-Flink Ingestion at Scale

Netflix’s graph ingestion pipeline turns member events into deduplicated node and edge updates with Kafka, Flink and Data Mesh.

– –
Event streams split into Flink processing lanes, then converge into graph node and edge records for shared data storage.
Conceptual illustration of the system described in the cited engineering account.

A graph becomes useful when applications need to traverse relationships that are difficult to reconstruct from isolated databases and warehouse tables. The harder problem is keeping those relationships current as events arrive from many product systems.

Netflix’s first post in its Real-Time Distributed Graph series describes an ingestion and processing pipeline for that challenge. [1] The company says it wanted internal consumers to query connected data across business areas that had evolved in separate services and stores. Part 1 focuses on the event path that turns member activity into graph nodes and edges; it does not cover the full storage or serving architecture.

From product events to graph updates

Netflix describes app events entering through an API gateway and being written to Kafka topics. The streams give downstream processors durable records they can consume and replay. [3] Topic data is encoded in Avro, with schemas held in an internal registry, and retention is tailored to each topic’s throughput and record size. The article also says records are persisted to Iceberg tables so teams can backfill when older records have expired from Kafka.

Flink jobs consume those topics and transform event records into graph primitives. The processing described includes filtering and projecting fields, enriching with side inputs, translating the event into one or more entity nodes and relationship edges, and publishing the resulting records to Data Mesh.

The publisher reports that some individual topics produce up to roughly one million messages per second and that the pipeline writes more than five million combined node and edge records per second downstream. These figures describe Netflix’s system as reported in the post; they should not be interpreted as a benchmark for every graph workload.

Why split the jobs by topic

Netflix first tried one Flink job for all source topics. The company says differences in volume and daily throughput made CPU, memory, parallelism and checkpoint configuration difficult to tune together. It moved to a one-to-one mapping between source topic and consuming job.

This creates more jobs to deploy and support, but isolates workload profiles. Teams can tune a high-volume event stream without forcing the same settings on a small, latency-sensitive topic. Netflix made a related choice to use separate topics for each node and edge type, accepting more Kafka configuration in exchange for more specific scaling and processing.

Coalesce overlapping changes before writing

Member activity can update the same graph entities repeatedly in a short period. Netflix describes stateful process functions and timers that buffer and deduplicate overlapping node and edge changes within a configurable time window. [2] This reduces the number of records written downstream while preserving the latest relevant state for that window.

The technique introduces a trade-off: aggregation reduces write volume but delays publication until the window or timer condition is met. The right interval depends on how quickly consumers need updates and how much duplicate work arrives. Teams should monitor event age as well as throughput so a lower downstream write rate does not hide stale state.

What this post does—and does not—show

The architecture offers a useful pattern for event-derived graphs: use Kafka as a replayable ingestion boundary, retain schemas and backfill data, separate stream jobs by workload, and coalesce repeated entity updates before persistence.

Netflix’s post explicitly treats graph storage and serving as separate layers in later installments. That scope matters: this first installment is evidence for the ingestion pipeline, not for the performance of graph traversals or the behavior of a particular storage engine. Start with the event contract, update semantics and recovery plan before choosing how to query the resulting graph.

Replay and state are part of the graph contract

A real-time pipeline must also recover from delayed, duplicate and out-of-order events. Kafka retention gives a processor a replay window; the Iceberg copy described by Netflix adds a path for backfills after a topic has expired older records. Those two stores serve different operational roles, and a rebuild needs stable schemas and a plan for reconciling historical and current graph state.

Deduplication windows trade freshness for fewer writes. A short window reduces latency but may miss updates that arrive later; a longer window combines more changes but delays downstream visibility and consumes more state. Operators should watch checkpoint duration, state size, late-event rates and the age of the newest graph update—not just records per second.

The one-job-per-topic design also makes ownership and deployment important. A new source topic requires a corresponding processor, schema compatibility checks and an update policy for the node or edge types it produces. This is additional operational work, but it prevents unrelated workload changes from forcing one global Flink configuration.

Advertisement

Frequently asked questions

What does Netflix’s Real-Time Distributed Graph represent?

Netflix describes a graph of entities and relationships that supports internal queries across data that had been split among business systems. Part 1 focuses on the pipeline that creates and updates graph records from events.

Why did Netflix use a separate Flink job for each Kafka topic?

The post says a single job was difficult to tune because topics had different volumes and throughput patterns. A one-to-one topic/job mapping made each job simpler to maintain and tune, at the cost of more jobs and topics to operate.

How does the ingestion pipeline reduce repeated updates?

Netflix describes stateful Flink functions and timers that buffer, detect and deduplicate overlapping updates to the same nodes or edges within a configurable window before publishing downstream.

Does Part 1 explain the graph’s storage and query serving layers?

No. The Netflix post explicitly limits Part 1 to ingestion and processing. Storage and serving are covered separately in later installments, so this article does not attribute Cassandra traversal or query-latency claims to Part 1.

Sources & further reading

/* Comments */