RabbitMQ vs Kafka vs Pulsar: Storage Engines, Retention Policies, and Delivery Guarantees
An architectural comparison of RabbitMQ, Apache Kafka, and Apache Pulsar: queue vs log storage, partition rebalancing, and tiering.
On this page
When engineering distributed architectures, asynchronous message brokers are the circulatory system of the microservices topology. They decouple producers from consumers, smooth out traffic spikes, and buffer critical state during downstream outages.
Yet, technical teams frequently engage in misguided debates asking: “Which message broker is fastest: RabbitMQ, Apache Kafka, or Apache Pulsar?”
Velocity is a function of workload shape. These three platforms were designed around fundamentally different storage models, retention semantics, and consumption paradigms:
+-------------------------------------------------------------------------------+
| THREE DISTINCT MESSAGING TOPOLOGIES |
+-------------------------------------------------------------------------------+
1. RabbitMQ: The Smart Broker, Dumb Consumer (Queue-Based)
Producer ---> Exchange ---> Queues ---> Consumer (Competes, Acks)
* Message Lifecycle: Ephemeral (Deleted immediately upon consumer ACK).
* Paradigm: Task distribution and complex routing.
2. Apache Kafka: The Dumb Broker, Smart Consumer (Distributed Log)
Producer ---> Append-Only Partitions ---> Consumer (Tracks Offset)
* Message Lifecycle: Durable retention (Persists for days/weeks).
* Paradigm: High-throughput event streaming and replayability.
3. Apache Pulsar: The Decoupled Compute-Storage Architecture
Pulsar Broker (Stateless Compute) <---> Apache BookKeeper (Segmented Storage)
* Message Lifecycle: Tiered storage (Streams to S3/cold storage).
* Paradigm: Unified queuing, streaming, and multi-tenant isolation.
1. RabbitMQ: The AMQP Queue-Centric Router
Built in Erlang on the Open Telecom Platform (OTP), RabbitMQ implements the Advanced Message Queuing Protocol (AMQP 0-9-1).
RabbitMQ operates on the principle of smart brokers and dumb consumers. The broker is responsible for routing logic, filtering, priority ordering, and tracking individual message delivery states.
+-------------------------------------------------------------------------------+
| RABBITMQ ROUTING ARCHITECTURE |
+-------------------------------------------------------------------------------+
[Producer]
|
v (Publishes message with Routing Key: "order.eu.created")
[Exchange (Direct / Topic / Fanout / Headers)]
|
+--- Binding: "order.eu.*" ---> [Queue: EU Orders] ---> [Worker A] (Acks & Deletes)
|
+--- Binding: "order.#" ---> [Queue: Audit Log] ---> [Worker B] (Acks & Deletes)
Message Lifecycle in RabbitMQ
- A producer sends a message to an Exchange.
- The Exchange inspects routing keys and header bindings, copying the message into matched Queues.
- Competing workers pull or receive messages via push subscriptions.
- When a worker acknowledges receipt (
basic.ack), RabbitMQ immediately deletes the message from memory and disk.
Best Suited For
- Complex, granular routing rules (topic wildcards, header matching).
- Individual message acknowledgments and immediate dead-letter queuing.
- Priority queues (processing high-tier customer jobs before standard jobs).
- Workloads where data does not need to be replayed once processed.
2. Apache Kafka: The Distributed Append-Only Commit Log
Apache Kafka rejected the queue abstraction entirely. In Kafka, there are no individual queues with message acknowledgment tracking.
Kafka models messaging as an append-only distributed commit log.
+-------------------------------------------------------------------------------+
| APACHE KAFKA LOG ARCHITECTURE |
+-------------------------------------------------------------------------------+
Topic: "user-signups" (Partition 0)
Offset: 0 1 2 3 4 5 6 7 8 ...
Record: [Msg] [Msg] [Msg] [Msg] [Msg] [Msg] [Msg] [Msg] [Msg]
^ ^ ^
| | |
Retention Head Consumer Group A Consumer Group B
(Oldest retained msg) (Offset: 4) (Offset: 8 - Near Real-Time)
The Log Paradigm
- Sequential Disk I/O: Kafka bypasses random disk seek times by writing messages sequentially to append-only segment files, taking full advantage of the Linux kernel Page Cache and zero-copy
sendfilenetwork system calls. - Consumer-Managed Offsets: The broker does not track whether individual messages were consumed. Consumers maintain their own offset pointers. Consuming a message does not delete it.
- Durable Retention: Messages persist for hours, weeks, or indefinitely based on retention policies. If an analytics service crashes or deploys a buggy algorithm, developers can reset the consumer offset to zero and replay months of production event history.
Best Suited For
- Massive horizontal throughput (millions of events per second per cluster).
- Event Sourcing, audit logs, and stream processing (Apache Flink, Kafka Streams).
- Pub-sub topologies where multiple independent systems consume the exact same event stream at different speeds.
3. Apache Pulsar: Decoupling Compute from Storage
While Kafka revolutionized streaming, its monolithic broker architecture carries operational friction: each Kafka broker couples compute (handling client connections) with persistent storage (local partition disks).
When a Kafka partition expands, rebalancing requires moving gigabytes of physical data across the network, saturating replication bandwidth.
Apache Pulsar solves this by enforcing a two-layer disaggregated architecture:
+-------------------------------------------------------------------------------+
| APACHE PULSAR TWO-TIER TOPOLOGY |
+-------------------------------------------------------------------------------+
[Producer Fleet]
|
v
+---------------------------------------------------------------+
| Stateless Broker Layer (Pulsar Brokers) |
| - Handles client connections, routing, and subscription state |
| - Zero persistent state on local disks; scales elastically! |
+---------------------------------------------------------------+
|
v (Stripes ledger segments across storage bookies)
+---------------------------------------------------------------+
| Distributed Storage Layer (Apache BookKeeper) |
| [Bookie 1: Seg 1] [Bookie 2: Seg 2] [Bookie 3: Seg 3] |
+---------------------------------------------------------------+
|
v (Asynchronous Background Tiering)
[Tiered Cold Storage: Amazon S3 / Google Cloud Storage]
The BookKeeper Segment Advantage
Pulsar partitions are divided into small segments (ledgers) distributed across a pool of storage nodes (Bookies).
- Instantaneous Scaling: Adding a storage Bookie or a stateless Pulsar broker requires zero data rebalancing; new segments are simply routed to the new node.
- Unified Queuing and Streaming: Pulsar supports multiple subscription modes: Exclusive (like Kafka), Failover, and Shared (competing consumers like RabbitMQ).
- Native Tiered Storage: Pulsar offloads aged segments directly to cloud object storage (S3), providing infinite retention at commodity pricing without disk exhaustion.
Architectural Comparison Matrix
| Feature | RabbitMQ | Apache Kafka | Apache Pulsar |
|---|---|---|---|
| Core Abstraction | AMQP Queues & Exchanges | Distributed Append-Only Log | Segmented Log + Disaggregated Storage |
| Storage Engine | Mnesia + Disk index (Khepri) | Sequential segment files | Apache BookKeeper Ledgers |
| Consumption Model | Broker pushes to consumers | Consumers pull by offset | Unified: Pull or Push subscriptions |
| Message Deletion | Deleted immediately upon ACK | Retained by time/size policy | Retained by policy; tiered to S3 |
| Message Replay | Not supported (Ephemeral) | Native (Reset offset pointer) | Native (Reset subscription cursor) |
| Throughput Ceiling | Tens of thousands msgs/sec | Millions of msgs/sec | Millions of msgs/sec |
| Rebalancing Cost | Low (Queues migrate easily) | Very High (Copies partition data) | Zero (Stateless broker additions) |
| Routing Complexity | Highly complex (Topic/Header) | Partition key hashing only | Partition key hashing only |
Decision Framework: Which Broker to Deploy?
DO YOU NEED TO REPLAY PAST EVENTS?
|
+-----------------------+-----------------------+
| YES | NO
v v
DO YOU NEED NATIVE MULTI-TENANCY DO YOU NEED COMPLEX AMQP ROUTING,
OR TIERED STORAGE TO S3? PRIORITY QUEUES, OR INSTANT PURGES?
| | |
| YES | NO | YES
v v v
[APACHE PULSAR] [APACHE KAFKA] [RABBITMQ]
Summary and Key Takeaways
- Match the broker to the data lifecycle: Use RabbitMQ when messages are transient tasks to be acknowledged and discarded; use Kafka or Pulsar when messages represent an event log of record.
- Kafka scales throughput via sequential logs: Bypassing individual message acknowledgment and writing sequential append logs enables Kafka to stream millions of events per second.
- Pulsar eliminates partition rebalance pain: By decoupling stateless compute brokers from Apache BookKeeper storage nodes, Pulsar scales elastically without network data reshuffling.
- RabbitMQ excels at complex routing: Leverage AMQP exchange bindings and topic patterns when fine-grained routing and per-message dead-lettering are required.
- Replayability enables resilience: Log-based systems permit time travel; when microservices suffer bugs, resetting consumer offsets allows full data reprocessing without data loss.

/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.