Thursday, May 31, 2012

RabbitMQ Exchanges & Queues naming convention with Burrow.NET

- If you just need only 1 exchange and 1 queue, naming convention should not be a big concern. Even that, you may need exchanges and queues for different environments. As in my projects, I often have 3 environments: DEV, UAT and PROD and I've been using following conventions for exchanges and queues:

Exchanges : ProjectName.E.<ExchangeType>.<Environment>.MessageTypeName
Queue        : ProjectName.Q.<Environment>.<ConsumerName>.MessageTypeName

- The consumer name in the Queue convention is optional. It makes sence to have different consumer names for different queues of the same message when Fanout exchange is used. Because at that case, we'll have many clients interested in the same message. We cannot let these clients subcribe to the same queue because only 1 of them will get the message while they all want the same thing. Fanout exchange in that sittuation is a best choice since it will deliver the same copy of the message you publish to all queues bind to it.

- In below example, we'll use exchange type Direct and we won't need ConsumerName in the queue name. Then a single message type will have 1 exchange and 1 queue bind to it for every environment. As a result, with message type "RequestMessage" we'll have

  • ProjectName.E.Direct.DEV.RequestMessage
  • ProjectName.E.Direct.UAT.RequestMessage
  • ProjectName.E.Direct.PROD.RequestMessage

  • ProjectName.Q.DEV.RequestMessage
  • ProjectName.Q.UAT.RequestMessage
  • ProjectName.Q.PROD.RequestMessage

- Creating queues and exchanges for every single message classes is not a good way as we will polute the server with a long list of queues and exchanges. Indeed, we realized that way as a problem and we've got a few better options. We can use 1 exchange for all those message types and bind different queues to the same exchange by different routing keys. So if we publish a message type RequestA to the exchange with routing key "RequestA", the message will go the queue "ProjectName.PROD.Q.RequestA" for instance. In our project, we had a need to use Fanout exchange as I mentioned above so we created a wrapper class like below:

public class MessageWrapper
    public object MessageObject {get; set;}

- So we'll have only 1 queue for each client. The message will be deserialized perfectly with the real object as the one we publish so the client can base on the type of MessageObject and determine what to do.

- Back to our story, with this convention in mind, what we have to do is implementing the IRouteFinder:

public class MyCustomRouteFinder : IRouteFinder
    private readonly string _environment;
    public MyCustomRouteFinder(string environment)
        if (string.IsNullOrEmpty(environment))
            throw new ArgumentNullException(environment);
        _environment = environment;

    public string FindExchangeName<T>()
        return string.Format("ProjectName.E.{0}.{1}.{2}", GetExchangeTypeFor<T>().ToUpper(), _environment, typeof(T).Name);

    public string FindRoutingKey<T>()
        return typeof(T).Name;

    public string FindQueueName<T>(string subscriptionName)
        if (GetExchangeTypeFor<T>() == ExchangeType.Fanout && string.IsNullOrEmpty(subscriptionName))
            throw new ArgumentException("subscriptionName");

        var subscription = GetSubscriptionTextFor<T>(subscriptionName);
        return !string.IsNullOrEmpty(subscription)
               ? string.Format("ProjectName.Q.{0}.{1}.{2}", _environment, subscription, typeof(T).Name)
               : string.Format("ProjectName.Q.{0}.{1}", _environment, typeof(T).Name);

    protected virtual string GetExchangeTypeFor<T>()
        var messageType = typeof(T);
        var messageTypesNeedDirectExchange = new List<Type> { typeof(MessageA), 
                                                              typeof(MessageB) };

        return messageTypesNeedDirectExchange.Contains(messageType)
             ? ExchangeType.Direct
             : ExchangeType.Fanout;

    protected virtual string GetSubscriptionTextFor<T>(string subscriptionName)
        var typesDontNeedSubscriptionName = new List<Type> { typeof(MessageA), 
                                                             typeof(MessageB) };

        return typesDontNeedSubscriptionName.Contains(typeof(T)) 
             ? null 
             : subscriptionName;

- Here is how to use it:

// Init
var environment = "DEV";
var routeFinder = new MyCustomRouteFinder(environment);
Global.DefaultSerializer = new JsonSerializer();

// Setup exchange and queues programatically
Func<string, string, IRouteFinder> routeFinderFactory = (environment, exchangeType) => routeFinder;
var setup = new RabbitSetup(routeFinderFactory, 
setup.SetupExchangeAndQueueFor<MessageA>(new ExchangeSetupData(), new QueueSetupData());
setup.SetupExchangeAndQueueFor<MessageB>(new ExchangeSetupData(), new QueueSetupData());

// Publish message 
using(var tunnel = RabbitTunnel.Factory.Create())
    tunnel.Publish<MessageA>(new MessageA());
    tunnel.Publish<MessageB>(new MessageB());

- And I swear by the moon and the stars in the sky that the messages will go to correct queues. In my last post, I'd mentioned the ConstantRouteFinder which can be used for a predefined exchange name, queue. However, in any cases, I think having a convention is better eventhough we have only 1 queue and exchange.

- As a best practice, we should not share the development environment with PROD or UAT environment so the "DEV" exchanges and queues should not be created on the same server as exchanges and queues for "UAT" and "PROD". We should install rabbitmq server on personal machine for development, it won't take so long.

Happy messaging.


Bob Sturm said...

A cleaner approach for sharing a broker instance across multiple lower environments is to use virtual hosts. This allows you to keep the exchange and queue names consistent across environments.

Van Nguyen said...

You're absolutely right. However, sometime people need the same data from PROD environment and to run test on UAT environment, at this case I don't think we can bind queues from a virtual host to another exchange on different virtual host ;).

Unknown said...

UAT / PROD / DEV environments should be isolated. Adding the environment into the queue name indicates a break in this separation.

Van Nguyen said...

Depend on use cases, in our use cases, we processed tweets so it doesn't matter.

Post a Comment