Ideas
Kafka is not a JMS
Kafka is a distributed log with its own concepts: partitions, offsets, consumer groups, retention. Teams that bring their message-queue reflexes to it get it wrong, so the concepts come first
Among the recommendations I wrote for a Kafka platform earlier this year, one fits on a single line.
Kafka is not a JMS. It is a technology of its own, with its own concepts
The trap is that it looks like one. There are producers, consumers, topics and messages. A team that has spent years on a message queue recognises the words and brings its habits along. That is where it breaks.
In a queue, a consumed message is gone: the broker tracks who received what and deletes what was read. In Kafka, a topic is a log. A message stays as long as the topic’s retention allows, read or not, and each consumer group keeps its own position in that log, its offset. Reading removes nothing. A group can rewind and read again, and a new group can arrive later and start from the beginning.
So the questions change. Two services must receive the same message? That is not routing: give each its own group. The third instance sits idle? That is a partition question: within a group, each partition goes to one consumer, so there are never more busy consumers than partitions. Is the order guaranteed? Only within a partition, which makes the partitioning key a design decision. When is the message deleted? When retention says so, a choice made per topic, not a side effect of reading.
None of this is hard. It is different, and the cost comes from treating it as the same: topics sized like queues, partition counts set without counting the consumers, retention left at its default, an order assumed where nothing promised one.
The order I recommend runs against instinct. Learn the four concepts before writing the first consumer: partitions, offsets, consumer groups and retention.
Only then sort through the old habits. Fewer survive than you expect