Scheduled Message Delivery with RabbitMQ

Earlier this month I gave a presentation at ComoRichWeb on RabbitMQ and one question from an attendee was “Is it possible to publish a message to be consumed at a later date?” I answered that it wasn’t possible to the best of my knowledge, but that there might be some hack to accomplish it. Well, this evening while trying to figure out how to use a push vs. polling model for timed notifications I discovered a clever hack using temporary queues, x-message-ttl and dead letter exchanges.

The main idea behind this is utilizing a new feature available in 2.8.0, dead-letter exchanges. This AMQP extension allows you to specify an exchange on a queue that messages should be published to when a message either expires or is rejected with requeue set to false.

With this in mind, we can simply create a queue for messages we want to be delivered later with an x-message-ttl set to the duration we want to wait before it is delivered. And to ensure the message is transferred to another queue we simply define the x-dead-letter-exchange to an exchange we created (in this case I’ll call it immediate) and bind a queue to it (the “right.now.queue”).

In coffeescript with node-amqp this looks like this:

Next I define the immediate exchange, bind a queue to it and subscribe.

Finally, after defining the queue I created earlier we want publish a message on it. So to revisit the earlier queue definition we add a publish call to publish directly to the queue (using the default exchange).

The result of running this is we’ll see a 5 second wait and then the message content and headers get dumped to the console. Since the queue is only used temporarily in this scenario I also set the x-expires attribute of the queue to expire in a reasonable amount of time after the message expires. This makes sure we don’t wind up with a ton of unused queues just sitting around.

Here’s the result of this exercise in its entirety.

You can get this exercise in full on github.

This is pretty interesting and I plan to experiment further with utilizing this in one of my production node.js applications that use interval based polling to trigger scheduled events.

  • http://tuple23.com adam

    Neat.

    When is a message determined to be “dead-letter”/expired? That is, if I have a message sitting in an idle queue, and the ttl passes, will the message be immediately dead-lettered? Or will it only be evaluated when it comes to the head of the queue?

  • http://blog.james-carr.org James Carr

    AFAIK the message will be dead-lettered as soon as the TTL for it has elapsed, queue position doesn’t have any bearing on whether or not it is expired.

  • http://jonathanoliver.com Jonathan Oliver

    That’s awesome! I’ve been hoping that RabbitMQ would build this into their system for a while now. This is a very creative and a very effective mechanism to support “timeout” windows.

    One comment that I would add is that I would recommend adding an additional header to a given message to distinguish a true message TTL and this technique you’ve come up with.

  • Fabrizio Sestito

    Hi,
    I tried to run your example but the messages are dropped after the ttl expires.
    If I create the queues and the exchange in the management panel and I try to publish a message in the dlx queue (without expires this time) I get the same behaviour. The message just disappears.
    Tested on Mac OS X with rabbitmq 2.8.7 and on ubuntu with 3.0.0.

    Am I missing something?

  • http://andreasohlund.net Andreas Öhlund

    I did a spike and hit a few showstoppers:

    1. Messages are only DLQ:en when at the top of the Q (http://www.rabbitmq.com/ttl.html – Caveats section)
    This means that if I first set msg 1 to expire in 4 hours and msg2 to expire in 1 hours msg2 will only expire after msg1 has expired.

    2. The TTL for the message is kept by Rabbit so lets say you use a short timeout of 10 s. If the consumer hasn’t been able to consume the message withing 10 seconds after it expired (due to a backlog) it will be discarded and lost

    The above has been verified with Rabbit 3.0.1 on windowss

    Do you guys see any workarounds?

  • Adam Mills

    He appears to be creating a queue per message