Why did Netflix use a separate Flink job for each Kafka topic?
The post says a single job was difficult to tune because topics had different volumes and throughput patterns. A one-to-one topic/job mapping made each job simpler to maintain and tune, at the cost of more jobs and topics to operate.
Answered in
Inside Netflix’s Real-Time Graph: Kafka-to-Flink Ingestion at ScaleNetflix’s graph ingestion pipeline turns member events into deduplicated node and edge updates with Kafka, Flink and Data Mesh.
Read the full analysisOther questions this article answers
More data engineering questions
- What does Netflix’s Real-Time Distributed Graph represent?
- How does the ingestion pipeline reduce repeated updates?
- Does Part 1 explain the graph’s storage and query serving layers?
- How does Spotify describe its data platform?
- What tools appear in Spotify’s published architecture?
- How many events does Spotify report handling?
- What is Scio?
- Was Zalando’s article a report of a fully automated production rollout?
Every answer on Crashtech is written by the editor of the article it comes from — never auto-summarised. Browse all answers or the Data Engineering beat.