How Nubank’s In-House Log Platform Separates Ingestion, Storage, and Search
Nubank describes a Fluent Bit and micro-batch pipeline feeding Parquet on S3, queried with Trino, after its vendor-coupled logging stack became hard to operate.
On this page

When a production incident begins, missing logs are more than an analytics inconvenience: engineers may not be able to reconstruct what happened. Nubank says its earlier logging setup coupled application forwarders directly to an external vendor. The company found gaps in end-to-end retention, limited visibility into ingestion, growing costs, and difficulty changing how data was filtered or queried. Its response was not one replacement database. It split the system into an ingestion stream and a separate storage-and-query platform. [1]
Start with the failure mode, not the storage format
Nubank’s post says the old arrangement sent application logs directly to the vendor and also allowed other internal sources to send data to the vendor API. That architecture had worked, but the company describes three constraints as volumes grew: the ingestion path was hard to observe, alerts and dashboards were tied to the vendor interface, and a surge in ingest could affect query performance because the two processes were coupled. [1]
Those problems are related but not identical. Missing or delayed records point to reliability and delivery visibility. A per-volume contract creates cost uncertainty. Vendor-specific dashboards and query syntax increase migration effort. A well-designed replacement must measure and improve these properties separately; moving the same tightly coupled pipeline onto a different storage service would not automatically solve them.
Nubank divided the program into two parts: an Observability Stream to receive and process signals, then a Query and Storage Platform for retention and search. Building the stream first let the company begin migration without forcing developers to change their logging experience all at once, while also giving the team metrics to inform later storage decisions. [1]
Micro-batching creates a clear ingestion boundary
The company describes three components in the stream. Fluent Bit collects and forwards data. A Data Buffer Service accumulates incoming records into larger chunks for a micro-batching architecture. An in-house Filter and Process Service applies filtering and processing logic and collects metrics about the incoming stream. Nubank says this separation made it easier to extend processing and decouple ingestion from queries. [1]
A buffer between forwarders and storage can absorb short bursts and reduce the overhead of handling every tiny record as an independent write. It also creates a boundary where operators can measure queues, lag, rejection, and data loss. Those measurements matter: a large buffer can smooth spikes, but it can also hide a downstream outage until the queue grows. Teams copying the pattern should define how long the buffer may retain data, what happens at capacity, how retries are made idempotent, and which metrics alert before a retention gap appears.
Nubank’s public description does not name Kafka as a required component. Its architecture lists a Data Buffer Service and an internal Filter and Process Service, so it is more accurate to discuss the function of those stages than to substitute a familiar open-source broker that the source does not identify. [1]
Object storage and columnar files change the query trade-off
For the storage and search layer, Nubank says it chose S3 for storage, Parquet for the file format, and Trino as the query engine. The company describes a Parquet generator that turns processed log data into columnar files. Parquet organizes values by column within row groups and stores metadata that can help query engines avoid reading unnecessary data. The format specification and documentation explain this layout; they do not guarantee a particular compression ratio or query speed for every dataset. [1] [2]
Trino provides distributed SQL execution across data sources. Nubank says partitioning was important because queries can target relevant subsets rather than scanning the full collection. In an object-storage design, file sizing, partition keys, metadata, and compaction all shape the actual read cost. Too many tiny files can create planning and request overhead; overly broad partitions can force large scans. The workload should determine how data is divided, and query plans should be checked against real incident-response questions. [1] [3]
This storage model separates durable data from the compute used to query it. It can be attractive when organizations need long retention and can tolerate some query latency, but it is not automatically a drop-in replacement for a low-latency indexed search product. Searchability depends on table layout, metadata, engine capacity, and operational tuning. A platform team should benchmark the queries engineers actually use during incidents, not only a bulk scan or an attractive file-compression statistic.
Treat the numbers as a dated company report
In its September 2025 post, Nubank said the platform was its default for log storage and querying since mid-2024. The company reported one trillion logs ingested per day, about one petabyte per day, 45 days of searchable retention, almost 15,000 queries per day, and 150 petabytes scanned daily. It also said its benchmarks showed roughly 50 percent lower cost than market solutions. These are figures reported by Nubank; the post does not provide enough benchmark methodology to independently validate or generalize them. [1]
The distinction matters for readers considering a similar build. A vendor invoice is only one line in the total cost of ownership. An in-house platform also needs engineering time, on-call support, data-quality monitoring, security controls, capacity planning, and migration work. The financial comparison should include the cost of incomplete data, query latency, and operational incidents as well as storage and licenses.
A practical checklist before building your own
First, measure the existing pipeline end to end. Track what is emitted, accepted, buffered, processed, stored, and queryable. Then separate the requirements for collection, transformation, retention, and search. Decide how quickly a record must become searchable, how long it must be retained, and which queries need low latency.
Prototype on representative logs and incident questions. Test small and large partitions, late-arriving records, schema changes, partial failures, and replay. Validate that privacy-sensitive fields are removed or protected before durable storage. Keep the developer-facing logging interface stable while the backend changes, and compare old and new output during migration so a missing event does not look like a successful cutover.
Nubank’s case is useful because it connects architecture choices to an operating problem: the team wanted more control over data flow and cost, and it separated ingest from search to gain that control. Its specific volumes and savings belong to its own environment. For another organization, the right answer depends on query latency, retention, staffing, reliability targets, and the actual price of its current system. [1]
Frequently asked questions
How many logs did Nubank say its platform handles?
In a September 2025 engineering post, Nubank said its in-house platform ingested one trillion logs per day, totaling about one petabyte daily, with 45 days of searchable retention. These are company-reported figures for that post, not an independently audited current measurement.
What components did Nubank describe in its ingestion stream?
The post describes Fluent Bit forwarders, a Data Buffer Service that accumulates large chunks for micro-batching, and an in-house Filter and Process Service. It does not describe Kafka as a required part of this pipeline.
Why did Nubank use Parquet and Trino with S3?
Nubank describes Parquet for columnar log storage, S3 for object storage, and Trino for partitioned SQL queries. This separates durable storage from query execution and lets engineers search selected data without keeping every log in an always-indexed search cluster.
What cost saving did Nubank report?
Nubank’s post says the platform cost about 50 percent less than market solutions according to its benchmarks. The article does not publish a full comparison methodology, so treat that as a company-reported result rather than a guaranteed saving for other teams.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.