What components did Nubank describe in its ingestion stream?
The post describes Fluent Bit forwarders, a Data Buffer Service that accumulates large chunks for micro-batching, and an in-house Filter and Process Service. It does not describe Kafka as a required part of this pipeline.
Answered in
How Nubank’s In-House Log Platform Separates Ingestion, Storage, and SearchNubank describes a Fluent Bit and micro-batch pipeline feeding Parquet on S3, queried with Trino, after its vendor-coupled logging stack became hard to operate.
Read the full analysisOther questions this article answers
More development best practices questions
- What is the primary security vulnerability of storing JWTs in localStorage?
- How do server-side sessions scale horizontally across multiple instances?
- Why is immediate revocation nearly impossible with stateless JWTs?
- What is the recommended modern authentication architecture for SaaS apps?
- What is the N+1 query problem in GraphQL and how is it fixed?
- How does GraphQL Federation allow multiple backend teams to maintain one API?
- Why is caching harder in GraphQL than in traditional REST APIs?
- When should an engineering team prefer REST or gRPC over GraphQL?
Every answer on Crashtech is written by the editor of the article it comes from — never auto-summarised. Browse all answers or the Development Best Practices beat.