---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 4 min
---

Kafka Fundamentals: Topics, Partitions, Offsets and Consumer Groups Explained

A distributed commit log, not a queue. How partitions give ordering and scale, how consumer groups share work, and the settings that decide durability.

– –

Kafka is easy to describe as “a message queue” and that description produces most of the mistakes. It is closer to a database table that only grows, which many readers scan at their own pace. Once that picture clicks, partitions, offsets and consumer groups stop being separate vocabulary.

The pieces, in order

  1. Message (record)

    The basic unit: a key, a value, optional headers and a timestamp. More like an event row than a request.

  2. Topic

    A named stream of messages, like a folder or table for one kind of event: orders, page-views.

  3. Partition

    A topic is split into partitions, and each partition is an ordered, append-only sequence. Every message in a partition gets a sequential offset. Partitions spread across brokers, which is where Kafka’s scale comes from.

  4. Producer

    Writes messages to a topic, batching them for throughput. The producer picks the partition: by hashing the key, or round robin if there is no key.

  5. Consumer and consumer group

    Consumers read from partitions and track their position by offset. Consumers that share a group id split the partitions among themselves, so a group collectively reads the topic once.

  6. Broker and cluster

    A broker is a Kafka server. A cluster holds many, and each partition is replicated across several brokers, with one leader serving reads and writes and followers copying it.

topic: orders (3 partitions)

partition 0: [0][1][2][3][4] ->           key "alice" always here
partition 1: [0][1][2] ->                 key "bob"   always here
partition 2: [0][1][2][3] ->              key "carol" always here
                  ^
                  consumer group "billing" reads at its own offsets

Why partitions decide everything

  • Ordering. Kafka guarantees order only within a partition. Same key, same partition, so all of one customer’s events stay in sequence. There is no order across partitions.
  • Parallelism. In a consumer group each partition is read by exactly one member. Six partitions can keep six consumers busy, a seventh waits idle.
  • Throughput. More partitions means more parallel writers and readers, spread across brokers.
Choose the partition count with care

You can add partitions later, but doing so changes which partition a key maps to, which breaks per-key ordering for existing keys. Plan for growth up front: pick a count that covers your expected peak consumer parallelism with headroom.

A log, not a queue

In a classic queue a message is gone once someone takes it. In Kafka it stays until retention (by time or size) expires. That one fact explains the platform’s strengths.

PropertyConsequence
Messages retained on diskNew consumers can start from the beginning, bugs can be fixed and data reprocessed
Consumers track their own offsetA slow consumer never slows the others
Many consumer groupsBilling, search indexing and analytics each read the full stream independently
Sequential disk writes and readsHigh throughput on ordinary hardware

Where there is one entry per key that matters (the latest address for each customer), log compaction keeps the most recent value per key instead of discarding by age.

Advertisement

Not losing data: three dials

Durability is configuration, not a property you get free.

LayerSettingMeaning
Replicationreplication.factor=3Three copies of every partition on different brokers
Produceracks=allProducer waits until all in-sync replicas have the message
Brokermin.insync.replicas=2Refuse writes if fewer than two replicas are in sync
ConsumerCommit offsets after processingA crash replays the message instead of skipping it

Together they mean an acknowledged message survives the loss of a broker. The price is a bit of latency, and the consumer side gives you at-least-once delivery: after a crash you may see a message twice, so processing must be idempotent. Kafka also supports transactions and idempotent producers for exactly-once pipelines inside Kafka, but end-to-end exactly-once into an external system still needs idempotent writes on your side.

What it is good for

Event streaming and pipelines the classic use
Move data between systems reliably and in order per key, decoupling producers from consumers.
Change data capture database to stream
Stream every row change from a database log so other systems can react or stay in sync.
Log and metric aggregation operational data
Collect logs and metrics from many services into one durable, replayable stream.
Event-driven services microservices
Services publish facts, others subscribe, nobody calls anybody directly.

It is a poor fit for a handful of messages per second, request-response patterns, or priority queues with per-message acknowledgement. A simpler broker is the better tool for those.

Do

Key your messages by the entity whose order matters, make consumers idempotent, monitor consumer lag per partition, and set replication, acks and min in-sync replicas explicitly.

Don't

Use random keys and expect ordering, auto-commit offsets before processing finishes, or run a cluster for traffic a database table and a worker would handle.

Takeaway

Think of Kafka as a replicated log with many independent readers. Partitions give you scale and per-key order, offsets give each consumer its own pace and the ability to replay, and replication plus acknowledgement settings determine whether a message survives a crash. Get the keys, partition count and idempotency right and the rest of the platform follows.

Advertisement

Frequently asked questions

Is Kafka a message queue?

Not quite. Kafka is a distributed append-only log. Consumers read by offset and messages stay until retention expires, rather than being deleted when read. That lets many independent consumers read the same data, and lets one consumer replay history, which a classic queue cannot.

What ordering does Kafka guarantee?

Order within a single partition only. Messages with the same key go to the same partition, so they stay in order relative to each other. There is no global order across partitions, so design around per-key ordering, such as per-customer or per-order.

How many consumers can read a topic in parallel?

Within one consumer group, at most one consumer per partition, so partition count caps parallelism. A topic with six partitions can keep six group members busy. Extra consumers sit idle. Separate groups each get a full copy of the stream.

How do I avoid losing messages in Kafka?

Use replication factor of at least three, set producers to wait for acknowledgement from all in-sync replicas, set a minimum in-sync replica count of two, and commit consumer offsets only after processing succeeds. Each of those closes a distinct loss window.

Sources & further reading

/* Comments */