Kafka Fundamentals: Topics, Partitions, Offsets and Consumer Groups Explained
A distributed commit log, not a queue. How partitions give ordering and scale, how consumer groups share work, and the settings that decide durability.
On this page
Kafka is easy to describe as “a message queue” and that description produces most of the mistakes. It is closer to a database table that only grows, which many readers scan at their own pace. Once that picture clicks, partitions, offsets and consumer groups stop being separate vocabulary.
The pieces, in order
- Message (record)
The basic unit: a key, a value, optional headers and a timestamp. More like an event row than a request.
- Topic
A named stream of messages, like a folder or table for one kind of event:
orders,page-views. - Partition
A topic is split into partitions, and each partition is an ordered, append-only sequence. Every message in a partition gets a sequential offset. Partitions spread across brokers, which is where Kafka’s scale comes from.
- Producer
Writes messages to a topic, batching them for throughput. The producer picks the partition: by hashing the key, or round robin if there is no key.
- Consumer and consumer group
Consumers read from partitions and track their position by offset. Consumers that share a group id split the partitions among themselves, so a group collectively reads the topic once.
- Broker and cluster
A broker is a Kafka server. A cluster holds many, and each partition is replicated across several brokers, with one leader serving reads and writes and followers copying it.
topic: orders (3 partitions)
partition 0: [0][1][2][3][4] -> key "alice" always here
partition 1: [0][1][2] -> key "bob" always here
partition 2: [0][1][2][3] -> key "carol" always here
^
consumer group "billing" reads at its own offsets
Why partitions decide everything
- Ordering. Kafka guarantees order only within a partition. Same key, same partition, so all of one customer’s events stay in sequence. There is no order across partitions.
- Parallelism. In a consumer group each partition is read by exactly one member. Six partitions can keep six consumers busy, a seventh waits idle.
- Throughput. More partitions means more parallel writers and readers, spread across brokers.
You can add partitions later, but doing so changes which partition a key maps to, which breaks per-key ordering for existing keys. Plan for growth up front: pick a count that covers your expected peak consumer parallelism with headroom.
A log, not a queue
In a classic queue a message is gone once someone takes it. In Kafka it stays until retention (by time or size) expires. That one fact explains the platform’s strengths.
| Property | Consequence |
|---|---|
| Messages retained on disk | New consumers can start from the beginning, bugs can be fixed and data reprocessed |
| Consumers track their own offset | A slow consumer never slows the others |
| Many consumer groups | Billing, search indexing and analytics each read the full stream independently |
| Sequential disk writes and reads | High throughput on ordinary hardware |
Where there is one entry per key that matters (the latest address for each customer), log compaction keeps the most recent value per key instead of discarding by age.
Not losing data: three dials
Durability is configuration, not a property you get free.
| Layer | Setting | Meaning |
|---|---|---|
| Replication | replication.factor=3 | Three copies of every partition on different brokers |
| Producer | acks=all | Producer waits until all in-sync replicas have the message |
| Broker | min.insync.replicas=2 | Refuse writes if fewer than two replicas are in sync |
| Consumer | Commit offsets after processing | A crash replays the message instead of skipping it |
Together they mean an acknowledged message survives the loss of a broker. The price is a bit of latency, and the consumer side gives you at-least-once delivery: after a crash you may see a message twice, so processing must be idempotent. Kafka also supports transactions and idempotent producers for exactly-once pipelines inside Kafka, but end-to-end exactly-once into an external system still needs idempotent writes on your side.
What it is good for
It is a poor fit for a handful of messages per second, request-response patterns, or priority queues with per-message acknowledgement. A simpler broker is the better tool for those.
Do
Key your messages by the entity whose order matters, make consumers idempotent, monitor consumer lag per partition, and set replication, acks and min in-sync replicas explicitly.
Don't
Use random keys and expect ordering, auto-commit offsets before processing finishes, or run a cluster for traffic a database table and a worker would handle.
Takeaway
Think of Kafka as a replicated log with many independent readers. Partitions give you scale and per-key order, offsets give each consumer its own pace and the ability to replay, and replication plus acknowledgement settings determine whether a message survives a crash. Get the keys, partition count and idempotency right and the rest of the platform follows.
Frequently asked questions
Is Kafka a message queue?
Not quite. Kafka is a distributed append-only log. Consumers read by offset and messages stay until retention expires, rather than being deleted when read. That lets many independent consumers read the same data, and lets one consumer replay history, which a classic queue cannot.
What ordering does Kafka guarantee?
Order within a single partition only. Messages with the same key go to the same partition, so they stay in order relative to each other. There is no global order across partitions, so design around per-key ordering, such as per-customer or per-order.
How many consumers can read a topic in parallel?
Within one consumer group, at most one consumer per partition, so partition count caps parallelism. A topic with six partitions can keep six group members busy. Extra consumers sit idle. Separate groups each get a full copy of the stream.
How do I avoid losing messages in Kafka?
Use replication factor of at least three, set producers to wait for acknowledgement from all in-sync replicas, set a minimum in-sync replica count of two, and commit consumer offsets only after processing succeeds. Each of those closes a distinct loss window.

/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.