When I first started learning Apache Kafka, I quickly realized that three terms appeared everywhere: topics, partitions, and offsets. At first glance, they seemed straightforward, but understanding how they work together was what finally made Kafka click.
Many tutorials explain these concepts separately, yet beginners often struggle to see how they fit into the bigger picture. In this article, we'll break them down using simple examples and follow the journey of a message from the moment it's produced until it's consumed.
By the end, you'll understand not only what topics, partitions, and offsets are, but also why Kafka relies on them to achieve scalability, reliability, and high throughput.
What Happens When a Producer Sends a Message?
Imagine you're building a weather application that continuously streams weather updates from cities around the world. Every few seconds, your application sends data such as:
{
"city": "Nairobi",
"temperature": 17.2,
"humidity": 75
}
The producer doesn't send this message directly to a consumer. Instead, it sends the message to Kafka.
The first destination inside Kafka is a topic.
Think of a topic as a category or channel where related messages are stored.
For example:
- weather
- payments
- orders
- transactions
- user-signups
If your application streams weather information, all weather events belong in the weather topic.
One important thing to remember is that a topic does not process messages. It simply organizes them.
Why Aren't Topics Enough?
Imagine your weather application grows significantly.
Instead of collecting data from seven cities, it now receives updates from every major city in the world.
Millions of messages begin arriving every minute.
If Kafka stored every message in one giant file, reading and writing would eventually become a bottleneck.
This is where partitions come in.
A topic is divided into one or more partitions.
`Weather Topic
+----------------------+
| Partition 0 |
+----------------------+
+----------------------+
| Partition 1 |
+----------------------+
+----------------------+
| Partition 2 |
+----------------------+`
Rather than storing every message in a single location, Kafka distributes messages across multiple partitions.
This distribution allows producers to write data in parallel while multiple consumers read data simultaneously, making Kafka incredibly scalable.
How Does Kafka Decide Which Partition to Use?
Kafka determines the destination partition in several ways.
If the producer supplies a key, Kafka hashes that key and consistently routes messages with the same key to the same partition.
For example:
Key: Nairobi
might always be written to Partition 1.
Key: Tokyo
could always be written to Partition 0.
This guarantees that all messages for a particular city remain in order.
If no key is supplied, Kafka balances messages across available partitions.
What Are Offsets?
Inside every partition, messages are stored sequentially.
Kafka assigns each message a unique number called an offset.
Partition 0
Offset 0 → Nairobi
Offset 1 → Tokyo
Offset 2 → Berlin
Offset 3 → Nairobi
Offset 4 → Tokyo
Why Offsets Matter
Offsets are one of Kafka's most powerful features.
Instead of deleting messages immediately after they're consumed, Kafka keeps them for a configurable retention period.
Consumers simply remember the last offset they processed.
Imagine a consumer has read:
Offset 0
Offset 1
Offset 2
If it crashes, Kafka doesn't lose track.
When the consumer restarts, it resumes from:
Offset 3
This makes Kafka highly fault tolerant.
It also allows applications to replay historical events whenever needed.
Bringing Everything Together
Let's follow a weather update through Kafka.
Producer
│
Weather Topic
│
Partition 1
Offset 42
│
Consumer
The producer sends a weather update.
Kafka places it inside the weather topic.
Using the message key, Kafka chooses Partition 1.
The message receives Offset 42.
A consumer subscribed to the topic reads the message and records that it has processed Offset 42.
If the consumer stops unexpectedly, it can later continue from Offset 43 without rereading older messages.
Final Thoughts
Understanding topics, partitions, and offsets is one of the biggest milestones when learning Apache Kafka.
Topics organize related streams of events.
Partitions distribute data so Kafka can scale efficiently.
Offsets allow consumers to keep track of exactly where they are, making recovery and replay possible without losing data.
Once these three concepts become clear, ideas such as consumer groups, replication, leader election, and fault tolerance become much easier to understand.
SOCIAL SHARE CARD GENERATOR