Ideas
Every consumer is idempotent
With Kafka, receiving the same message twice is the normal case, not the exception: every consumer has to be written so that a second delivery changes nothing
One line in the Kafka recommendations I wrote this year looks excessive on first reading: every consumer must be idempotent, because receiving the same message more than once is very likely.
A duplicate is taken for a rare event, something that happens when things break. It is not rare. It is the normal behaviour of the default contract.
A consumer reads a batch, processes it, then saves its position, its offset. If it stops between the processing and the save, or if its group rebalances at that moment, the consumer that takes over the partition starts again from the last saved offset and processes the same messages a second time. Kafka’s documentation calls this at-least-once, and it is the default guarantee. A producer that retries after a network error can also write the same message twice; the idempotent producer keeps that duplicate out of the log, but it does nothing for the consumer that stopped after processing.
A duplicate is not a failure of the platform. It is the other side of a guarantee that nothing is lost
So the question is not how to avoid the second delivery, but what the second delivery does. Three answers cover most cases. Write by key: an update keyed on the business identifier turns the copy into a mere rewrite. Remember what was done: store the identifiers of processed messages in the same transaction as the effect, and skip the ones already seen. Make the effect repeatable by nature: set a status rather than increment a counter.
Exactly-once does exist between Kafka topics, through transactions, but it stops at the edge of the database or the API the consumer writes to.
The test follows: replay the same message twice and check that the second pass changed nothing.
Without that test, the consumer may be hiding an error that only a duplicate will reveal