---
topic: system-design
author: Crashtech Editorial
published: Nov 17, 2026 · read: 4 min
---

RabbitMQ vs Kafka vs Pulsar: Storage Engines, Retention Policies, and Delivery Guarantees

An architectural comparison of RabbitMQ, Apache Kafka, and Apache Pulsar: queue vs log storage, partition rebalancing, and tiering.

– –

When engineering distributed architectures, asynchronous message brokers are the circulatory system of the microservices topology. They decouple producers from consumers, smooth out traffic spikes, and buffer critical state during downstream outages.

Yet, technical teams frequently engage in misguided debates asking: “Which message broker is fastest: RabbitMQ, Apache Kafka, or Apache Pulsar?”

Velocity is a function of workload shape. These three platforms were designed around fundamentally different storage models, retention semantics, and consumption paradigms:

+-------------------------------------------------------------------------------+
|                       THREE DISTINCT MESSAGING TOPOLOGIES                     |
+-------------------------------------------------------------------------------+

1. RabbitMQ: The Smart Broker, Dumb Consumer (Queue-Based)
   Producer ---> Exchange ---> Queues ---> Consumer (Competes, Acks)
   * Message Lifecycle: Ephemeral (Deleted immediately upon consumer ACK).
   * Paradigm: Task distribution and complex routing.

2. Apache Kafka: The Dumb Broker, Smart Consumer (Distributed Log)
   Producer ---> Append-Only Partitions ---> Consumer (Tracks Offset)
   * Message Lifecycle: Durable retention (Persists for days/weeks).
   * Paradigm: High-throughput event streaming and replayability.

3. Apache Pulsar: The Decoupled Compute-Storage Architecture
   Pulsar Broker (Stateless Compute) <---> Apache BookKeeper (Segmented Storage)
   * Message Lifecycle: Tiered storage (Streams to S3/cold storage).
   * Paradigm: Unified queuing, streaming, and multi-tenant isolation.

1. RabbitMQ: The AMQP Queue-Centric Router

Built in Erlang on the Open Telecom Platform (OTP), RabbitMQ implements the Advanced Message Queuing Protocol (AMQP 0-9-1).

RabbitMQ operates on the principle of smart brokers and dumb consumers. The broker is responsible for routing logic, filtering, priority ordering, and tracking individual message delivery states.

+-------------------------------------------------------------------------------+
|                       RABBITMQ ROUTING ARCHITECTURE                           |
+-------------------------------------------------------------------------------+

[Producer] 
    |
    v (Publishes message with Routing Key: "order.eu.created")
[Exchange (Direct / Topic / Fanout / Headers)]
    |
    +--- Binding: "order.eu.*" ---> [Queue: EU Orders] ---> [Worker A] (Acks & Deletes)
    |
    +--- Binding: "order.#"    ---> [Queue: Audit Log] ---> [Worker B] (Acks & Deletes)

Message Lifecycle in RabbitMQ

  1. A producer sends a message to an Exchange.
  2. The Exchange inspects routing keys and header bindings, copying the message into matched Queues.
  3. Competing workers pull or receive messages via push subscriptions.
  4. When a worker acknowledges receipt (basic.ack), RabbitMQ immediately deletes the message from memory and disk.

Best Suited For

  • Complex, granular routing rules (topic wildcards, header matching).
  • Individual message acknowledgments and immediate dead-letter queuing.
  • Priority queues (processing high-tier customer jobs before standard jobs).
  • Workloads where data does not need to be replayed once processed.

2. Apache Kafka: The Distributed Append-Only Commit Log

Apache Kafka rejected the queue abstraction entirely. In Kafka, there are no individual queues with message acknowledgment tracking.

Kafka models messaging as an append-only distributed commit log.

+-------------------------------------------------------------------------------+
|                       APACHE KAFKA LOG ARCHITECTURE                           |
+-------------------------------------------------------------------------------+

Topic: "user-signups" (Partition 0)
Offset:     0     1     2     3     4     5     6     7     8 ...
Record:   [Msg] [Msg] [Msg] [Msg] [Msg] [Msg] [Msg] [Msg] [Msg]
            ^                       ^                       ^
            |                       |                       |
      Retention Head           Consumer Group A        Consumer Group B
    (Oldest retained msg)      (Offset: 4)             (Offset: 8 - Near Real-Time)

The Log Paradigm

  • Sequential Disk I/O: Kafka bypasses random disk seek times by writing messages sequentially to append-only segment files, taking full advantage of the Linux kernel Page Cache and zero-copy sendfile network system calls.
  • Consumer-Managed Offsets: The broker does not track whether individual messages were consumed. Consumers maintain their own offset pointers. Consuming a message does not delete it.
  • Durable Retention: Messages persist for hours, weeks, or indefinitely based on retention policies. If an analytics service crashes or deploys a buggy algorithm, developers can reset the consumer offset to zero and replay months of production event history.

Best Suited For

  • Massive horizontal throughput (millions of events per second per cluster).
  • Event Sourcing, audit logs, and stream processing (Apache Flink, Kafka Streams).
  • Pub-sub topologies where multiple independent systems consume the exact same event stream at different speeds.

3. Apache Pulsar: Decoupling Compute from Storage

While Kafka revolutionized streaming, its monolithic broker architecture carries operational friction: each Kafka broker couples compute (handling client connections) with persistent storage (local partition disks).

When a Kafka partition expands, rebalancing requires moving gigabytes of physical data across the network, saturating replication bandwidth.

Apache Pulsar solves this by enforcing a two-layer disaggregated architecture:

+-------------------------------------------------------------------------------+
|                       APACHE PULSAR TWO-TIER TOPOLOGY                         |
+-------------------------------------------------------------------------------+

[Producer Fleet]
       |
       v
+---------------------------------------------------------------+
| Stateless Broker Layer (Pulsar Brokers)                       |
| - Handles client connections, routing, and subscription state |
| - Zero persistent state on local disks; scales elastically!   |
+---------------------------------------------------------------+
       |
       v (Stripes ledger segments across storage bookies)
+---------------------------------------------------------------+
| Distributed Storage Layer (Apache BookKeeper)                 |
| [Bookie 1: Seg 1]   [Bookie 2: Seg 2]   [Bookie 3: Seg 3]     |
+---------------------------------------------------------------+
       |
       v (Asynchronous Background Tiering)
[Tiered Cold Storage: Amazon S3 / Google Cloud Storage]

The BookKeeper Segment Advantage

Pulsar partitions are divided into small segments (ledgers) distributed across a pool of storage nodes (Bookies).

  • Instantaneous Scaling: Adding a storage Bookie or a stateless Pulsar broker requires zero data rebalancing; new segments are simply routed to the new node.
  • Unified Queuing and Streaming: Pulsar supports multiple subscription modes: Exclusive (like Kafka), Failover, and Shared (competing consumers like RabbitMQ).
  • Native Tiered Storage: Pulsar offloads aged segments directly to cloud object storage (S3), providing infinite retention at commodity pricing without disk exhaustion.

Architectural Comparison Matrix

FeatureRabbitMQApache KafkaApache Pulsar
Core AbstractionAMQP Queues & ExchangesDistributed Append-Only LogSegmented Log + Disaggregated Storage
Storage EngineMnesia + Disk index (Khepri)Sequential segment filesApache BookKeeper Ledgers
Consumption ModelBroker pushes to consumersConsumers pull by offsetUnified: Pull or Push subscriptions
Message DeletionDeleted immediately upon ACKRetained by time/size policyRetained by policy; tiered to S3
Message ReplayNot supported (Ephemeral)Native (Reset offset pointer)Native (Reset subscription cursor)
Throughput CeilingTens of thousands msgs/secMillions of msgs/secMillions of msgs/sec
Rebalancing CostLow (Queues migrate easily)Very High (Copies partition data)Zero (Stateless broker additions)
Routing ComplexityHighly complex (Topic/Header)Partition key hashing onlyPartition key hashing only

Decision Framework: Which Broker to Deploy?

                                  DO YOU NEED TO REPLAY PAST EVENTS?
                                               |
                       +-----------------------+-----------------------+
                       | YES                                           | NO
                       v                                               v
        DO YOU NEED NATIVE MULTI-TENANCY                 DO YOU NEED COMPLEX AMQP ROUTING,
        OR TIERED STORAGE TO S3?                         PRIORITY QUEUES, OR INSTANT PURGES?
           |                         |                                 |
           | YES                     | NO                              | YES
           v                         v                                 v
     [APACHE PULSAR]          [APACHE KAFKA]                     [RABBITMQ]

Summary and Key Takeaways

  1. Match the broker to the data lifecycle: Use RabbitMQ when messages are transient tasks to be acknowledged and discarded; use Kafka or Pulsar when messages represent an event log of record.
  2. Kafka scales throughput via sequential logs: Bypassing individual message acknowledgment and writing sequential append logs enables Kafka to stream millions of events per second.
  3. Pulsar eliminates partition rebalance pain: By decoupling stateless compute brokers from Apache BookKeeper storage nodes, Pulsar scales elastically without network data reshuffling.
  4. RabbitMQ excels at complex routing: Leverage AMQP exchange bindings and topic patterns when fine-grained routing and per-message dead-lettering are required.
  5. Replayability enables resilience: Log-based systems permit time travel; when microservices suffer bugs, resetting consumer offsets allows full data reprocessing without data loss.
Advertisement

/* Comments */