A consumer reads a topic of notifications and delivers each one to the client it is meant for. Three clients, one topic, one consumer group. It works until the day one client stops answering. The consumer retries the message for that client, and while it retries, nothing else moves: the messages for the two clients that are perfectly healthy wait behind it.
This is head-of-line blocking, and Kafka makes it the default. Kafka also offers no built-in way out. In Kafka 2.3, a producer cannot ask the broker to deliver a record later, and a consumer cannot set one record aside and acknowledge the next. A delay in Kafka is something you design. Here are three ways to do it, what each one costs, and when to pick which.
Why one slow client stops everyone
Kafka guarantees order within a partition, and a consumer group records its progress in each partition as a single number: the offset of the next record to read. There is no acknowledgement per message. Committing past M1 says that M1 is done. If it is not, it is lost.
So when the call for M1 fails, the consumer has two honest options: retry M1, or give up on it. Retrying keeps the order and blocks the partition. M2 and M3 are already fetched, their clients are up, and they wait.
Retrying forever hits a second limit. A consumer must call poll() within
max.poll.interval.ms, five minutes by default. If it does not, the
documentation is explicit: “the consumer is considered failed and the group will
rebalance in order to reassign the partitions to another member”. That member
then meets the same M1. A consumer that sleeps through a long outage does not
just wait, it causes rebalances.
The real question is not how to retry. It is where a message can wait without holding the others back, and the answer depends on where the wait belongs.
Design 1: one consumer group per client
The first design gives each client its own consumer group. Every group reads the whole topic, keeps the messages addressed to its client and skips the others. Group-1 only delivers to Client 1, Group-2 to Client 2, Group-3 to Client 3.
Groups do not share offsets. Each moves through the partition at its own pace, so when Client 1 is slow or down, only Group-1 waits. Group-2 and Group-3 have already skipped M1 as not theirs and delivered their own messages. The consumers are completely independent of each other: a slow or unavailable client delays its own notifications and nobody else’s.
What makes the design attractive is what it leaves unchanged. Group-1 can still retry M1 in place, with the same loop as before, because the only messages it holds back belong to the same client. Order per client is preserved, and there is no new topic, no new message format and no scheduler.
The cost lies in reads and in operations:
- Every group reads every message. With three clients the topic is read three times, and two thirds of each consumer’s work is deserialising records only to discard them. With thirty clients, that waste becomes the main cost.
- A new client means a new group, deployed and monitored. The list of clients lives in the deployment, not in the data.
- Each group scales like any other: to scale out, the topic needs at least as many partitions as the group has consumers.
In return, the lag of each group becomes a clean signal: a group falling behind names the client that is struggling.
Design 2: one delay topic for everyone
The second design keeps a single consumer group for all clients and gives the messages it cannot deliver a place to wait: a delay topic, shared by every client and every source topic.
When a client does not respond, the consumer does not retry. It wraps the message, writes it to the delay topic, and moves on to the next record. From then on, that client’s messages are delayed as well, while the others flow. The wrapper carries everything needed to put the message back later:
key: null
value: {
topic: notify-client,
endTime: timestamp + delay,
key: original key,
value: original value
}
The topic field is what lets one delay topic serve the whole platform: the
message knows where it must return. endTime is the moment it becomes due
again.
A second consumer group reads the delay topic, but not continuously. A schedule
starts it every minute, counted from the end of the previous run, so two runs
never overlap. Each run takes every record whose endTime has passed,
republishes it to its original topic with its original key and value, and stops
at the first record still in the future.
Stopping at the first record not yet due works because every message waits the
same delay. Records reach a partition in the order they failed, so their
endTime only grows along it. The run commits up to the last record it
republished and leaves the rest for the next minute. With a delay that varied
per message, a record due late would hold back records due earlier, and the
delay topic would block exactly like the topic it was meant to relieve. One
fixed delay is not a simplification. It is what keeps the design sound.
The design is cheap. One group serves every client, and one topic with one scheduled consumer serves every source topic. While Client 1 is down, its messages flow into the delay topic, and Client 2 and Client 3 are served without waiting.
It gives up two things.
Order. A delayed M1 comes back behind messages produced after it. The null key also spreads the delayed records across the partitions of the delay topic, so their relative order is gone too. If a client depends on receiving its messages in order, this design is not for that client.
An end. A client that stays down sends its messages around the loop again and again. The wrapper above has no way to stop that. In practice it needs a count of attempts and a limit, after which the message goes to a dead letter topic with its original key, its original value, its source topic and the reason it failed, where someone can look at it.
Delays are also coarse: a message waits the delay plus up to a minute for the next run. For notifications, that is usually acceptable. For anything with a deadline, measure it.
Design 3: tiered retry topics
A third design is public and well documented. Uber Engineering described it in February 2018 in Building Reliable Reprocessing and Dead Letter Queues with Apache Kafka. It is not one of the two designs above, but it answers the same question and is worth knowing before choosing.
The consumer of the main topic tries a message a set number of times. If it still fails, it publishes the message to a first retry topic and commits, so the batch moves on. Each retry topic has its own consumers, which wait for a delay before processing. A failure at one level moves the message to the next retry topic, with a longer delay, and the last topic in the chain is a dead letter queue. Messages can later be merged from the dead letter queue back into the first retry topic, away from live traffic. The post is careful about the guarantee it gives: “handler invocation will occur at least after the configured timeout but possibly later”.
Where the delay topic of design 2 holds one fixed delay and is drained on a schedule, tiered retry topics give a back-off: a quick second try for a transient error, longer waits for an outage. Each level still has a single delay, so each retry topic stays ordered by due time, for the same reason as above. The price is topics: a chain for each consuming application, times the number of levels.
Waiting inside a retry consumer brings back the max.poll.interval.ms problem.
The Java consumer has a tool for it: pause(). In the 2.3 javadoc, future calls
to poll() “will not return any records from these partitions” until
resume(), and pausing “does not cause a group rebalance when automatic
assignment is used”. The consumer keeps calling poll(), stays in its group,
receives nothing from the paused partition, and resumes once the record at its
head is due. A seek() to the offset of that record makes the next poll()
read it again.
pause() also handles another case on its own. When the dependency is shared, a
database or a single partner API, and it fails for every message alike, moving
messages to another topic gains nothing. Pausing the partition until the
dependency is back keeps the order and costs nothing.
Which one, when
- One consumer group per client when the clients are few and stable, and each one needs its messages in order. It costs reads, and a deployment per client.
- One delay topic for everyone when clients are many or change often, and order does not matter. It costs a wrapper format, a scheduled consumer and a limit on the loop, and it delays by a fixed amount, to the minute.
- Tiered retry topics when failures come in different kinds, transient and lasting, and deserve different waits. It costs a topic and a group per level, for each application.
- pause() and resume() when the whole downstream is down, not one client. Nothing moves, nothing is lost, the order holds.
Whatever you choose, consumers are idempotent
Each design delivers some messages twice. In design 1, a call that timed out may still have reached the client, and the retry sends it again. In designs 2 and 3, a run that republishes a message and stops before committing its offset republishes it once more on the next run.
So the recommendation I make to every team starting with Kafka holds in all three cases: every consumer must be idempotent, because receiving the same message more than once is very likely, not exceptional. Concretely, each message carries an identifier that survives the wrapper and the republication, and the client, or the consumer in front of it, remembers what it has already handled. A correlation id injected into the headers and propagated by every consumer makes the journey of a message through the delay loop traceable when someone has to explain why it arrived late.
Decide where the wait belongs
Before adding a topic, ask who should wait. If one client is slow and its order matters, give it its own group. If many clients come and go and order does not matter, give them one delay topic. If failures need a back-off, chain retry topics. If the whole downstream is down, pause. Kafka will not hold a message for later, so the design has to decide who waits, and for how long.