When Does a RabbitMQ Message Really Survive a Restart?

Ask a team whether a message can be lost and the usual answer is “the queue is durable”. It is a good start and a bad finish. In RabbitMQ, durability is not a property of a queue. It is the agreement of several settings along the path a message takes, and each of them covers a different failure.

A message travels from a producer to an exchange, from the exchange to one or more queues through bindings, and from a queue to a consumer. It can be lost in transit to the broker, inside the broker when it restarts, at the exchange when nothing routes it, and in the consumer when it crashes halfway through. Five settings close these gaps. Miss one and the others still look fine.

Enlarge

Close

1. A durable exchange and a durable queue

Exchanges and queues are declared with a durable flag. A durable one is stored by the broker and comes back after a restart; a transient one is gone.

channel.ExchangeDeclare("billing.prod.orders", ExchangeType.Direct,
                        durable: true);
channel.QueueDeclare("billing.prod.invoices", durable: true,
                     exclusive: false, autoDelete: false, arguments: null);
channel.QueueBind("billing.prod.invoices", "billing.prod.orders", "order.paid");

The two flags protect different things. A transient queue loses its messages at restart, even the persistent ones. A transient exchange loses itself and its bindings: the messages already stored in a durable queue stay, but the next publish to the missing exchange is a channel error that closes the channel. And if the application declares the exchange again at startup, the bindings are still missing: every message it publishes is now unroutable, which is the fourth setting’s business.

Changing your mind later has a cost. Redeclaring an existing queue with different attributes raises a 406 PRECONDITION_FAILED. A transient queue cannot be made durable in place, it has to be deleted and declared again. Decide before the first deployment.

2. A persistent message

The RabbitMQ documentation states the trap in one sentence: durability of a queue does not make the messages routed to it durable. After a restart, a durable queue is declared again, but only persistent messages are recovered. A message is persistent when it is published with delivery mode 2.

IBasicProperties props = channel.CreateBasicProperties();
props.DeliveryMode = 2; // persistent

Element	Survives a restart when	Otherwise
Exchange	it is declared durable	lost, with its bindings
Queue	it is declared durable (or quorum)	lost, with its messages
Message	it is published with mode 2	lost if the broker stops

Persistence is not immediate. The broker writes persistent messages to disk in batches, after a fraction of a second or when the queue is idle. A node that fails inside that window loses messages the producer believes are safe. Only the next setting tells the producer which ones.

3. Publisher confirms

BasicPublish returns nothing. The client writes a frame to a socket and moves on; the message may be lost on the way or delayed for a long time. In standard AMQP 0-9-1, the only guarantee is a transaction, which the RabbitMQ documentation measures as 250 times slower. Publisher confirms are the lightweight answer: once a channel is in confirm mode, the broker acknowledges each message it has taken responsibility for.

channel.ConfirmSelect();
channel.BasicPublish("billing.prod.orders", "order.paid", props, body);
channel.WaitForConfirmsOrDie(TimeSpan.FromSeconds(5));

When the ack comes depends on where the message went. For a persistent message in a durable queue, it comes after the write to disk. For a mirrored queue, it comes when every mirror has accepted the message; for a quorum queue, when a majority of replicas has. A basic.nack only happens on an internal error, and the producer may then publish again.

Waiting for each confirm in turn is safe and slow: under constant load, the documentation warns, an ack for a persistent message can take a few hundred milliseconds. A producer that needs throughput handles BasicAcks and BasicNacks asynchronously, tracking NextPublishSeqNo for each message, and republishes with a backoff when no ack arrives. A republished message may already be in the queue: the consumer has to accept duplicates.

4. The mandatory flag

Here is the case confirms do not cover. A message that no binding matches is unroutable. With mandatory set to false, the default, the broker discards it (or sends it to an alternate exchange, if one is configured). And it still confirms it: the broker issues the ack once the exchange has found that no queue will take the message. The producer receives its ack for a message that no longer exists.

A confirm says the broker took responsibility. It does not say a queue holds the message. The mandatory flag closes that gap: the broker returns the unroutable message with a basic.return, sent before the ack.

channel.BasicReturn += (sender, ea) =>
    log.Error($"Unroutable: {ea.Exchange}/{ea.RoutingKey} {ea.ReplyText}");

channel.BasicPublish("billing.prod.orders", "order.paid",
                     mandatory: true, basicProperties: props, body: body);

The handler matters as much as the flag: in the .NET client, a returned message with no listener on BasicReturn is silently dropped. Unroutable messages come from ordinary mistakes, such as a mistyped routing key, a binding created after the first publish, or a queue renamed in one service and not in the other. RabbitMQ 3.8 also counts unroutable dropped messages in its metrics, a count worth alerting on.

5. Manual consumer acknowledgements

The last gap is on the consumer side. In automatic acknowledgement mode, the broker considers a message delivered as soon as it has sent it. If the consumer crashes while processing it, or its connection drops, the message is lost. The documentation calls this mode unsafe, and it has a second cost: with no limit on unacknowledged messages, a slow consumer can be flooded until it runs out of memory.

channel.BasicQos(prefetchSize: 0, prefetchCount: 20, global: false);

var consumer = new EventingBasicConsumer(channel);
consumer.Received += (sender, ea) =>
{
    try
    {
        Process(ea.Body);
        channel.BasicAck(ea.DeliveryTag, multiple: false);
    }
    catch (InvalidOrderException)
    {
        channel.BasicNack(ea.DeliveryTag, multiple: false, requeue: false);
    }
};
channel.BasicConsume("billing.prod.invoices", autoAck: false, consumer);

With manual acks, the consumer acknowledges after processing, never before. A message not yet acknowledged when the channel or the connection closes is put back in the queue and delivered again with the redelivered flag set, possibly to another consumer. A message the consumer can never process is rejected with requeue: false, which sends it to a dead letter exchange when the queue has one. Every critical queue deserves one, with an alert on it.

Two consequences follow. The consumer must be idempotent, since a redelivery of a message it already processed is part of normal operation. And ordering is no longer guaranteed after a crash: a requeued message comes back while others have moved on. When order is essential, one consumer with a prefetch of 1 keeps it; otherwise, accept out-of-order processing. Size the prefetch to what the consumer can really handle.

Surviving the loss of a node

Everything so far keeps a message through a restart. It does not keep it through the loss of the disk it was written to. For that, the queue must be replicated across the nodes of a cluster, and in RabbitMQ 3.8 there are two ways to do it.

Enlarge

Close

Classic mirrored queues are configured by a policy (ha-mode). Each queue has a master and one or more mirrors; every operation goes through the master, which propagates it. A confirm waits for all the mirrors. When the master fails, the longest-running mirror is promoted. The trap is synchronisation: a mirror added to an existing queue starts empty and only receives new messages. By default (ha-promote-on-failure set to always), an unsynchronised mirror can be promoted, and the messages that only existed on the master are lost. The quorum queue guide itself gives the reason for a new type: in certain failure scenarios, mirrored queues can confirm messages too early.

Quorum queues arrived with RabbitMQ 3.8.0 in October 2019. They implement a durable, replicated FIFO queue on the Raft consensus algorithm: a leader and followers, a confirm once a majority of replicas has written the message, and a new leader elected among the followers when the leader is lost. A follower that comes back catches up only on the delta. With three replicas, the queue tolerates one failed node; with five, two. A two-node cluster loses its majority as soon as one node fails, which is why odd sizes are recommended. Three or five nodes is advice, not a limit: by default a quorum queue starts up to five replicas, one per node.

A quorum queue is chosen at declaration, with an argument; a policy cannot turn an existing queue into one.

channel.QueueDeclare("billing.prod.invoices", durable: true,
                     exclusive: false, autoDelete: false,
                     arguments: new Dictionary<string, object>
                     {
                         { "x-queue-type", "quorum" }
                     });

What a quorum queue cannot do in 3.8

The new type is purpose-built, and the 3.8 documentation lists its limits plainly.

• It is always durable and never exclusive: it is not meant for temporary queues.
• Every message is persisted, whatever its delivery mode.
• It supports no TTL, neither on the queue nor on messages, and no priorities.
• Queue length limits only apply with the drop-head overflow behaviour.
• Global QoS is refused; per-consumer prefetch works.
• Replicas are managed by hand: a node added to the cluster hosts none until an operator runs rabbitmq-queues add_member.

It also brings something mirrored queues lack: it counts delivery attempts in an x-delivery-count header, and a delivery-limit policy dead-letters a poison message instead of letting it loop forever.

The guide is clear about when to use them. Quorum queues are for long-lived queues that matter, such as incoming orders, used with publisher confirms and manual acknowledgements. Without both, the guarantees do not apply. My rule follows: a critical queue is a quorum queue, and a classic queue needs a justification, such as a temporary reply queue or a need for TTL.

The routing traps behind unroutable messages

The mandatory flag catches routing mistakes; knowing the exchange types prevents most of them.

• Fanout sends every message to every bound queue and ignores the routing key: leave it empty.
• Direct routes on an exact match between routing key and binding key. Bind two queues with the same key and each receives a copy, exactly like a fanout.
• The default exchange is a nameless direct exchange to which every queue is bound by its own name. Publishing to "" with a queue name works, until the queue is renamed. Name the exchange explicitly, and keep the default for the exceptional case.
• Topic matches dot-separated words: * stands for exactly one, # for zero or more. order.created.fr matches order.*.* and order.#, but not order.*.
• Headers routes on header values rather than the key; it is rarely needed.

The checklist

• Exchanges and queues declared durable before the first deployment.
• Critical messages published with delivery mode 2.
• Publisher confirms on, handled asynchronously, with a republish and a backoff when no ack arrives.
• mandatory: true on important messages, with a BasicReturn handler that does something.
• Consumers in manual ack mode, acking after processing, with a prefetch sized to their capacity, idempotent processing and a dead letter exchange.
• Quorum queues for what must survive the loss of a node, on an odd number of nodes, knowing what they cannot do yet.

A message survives when every link in the chain agrees. The durable queue was only the first.