Messaging with RabbitMQ and Burrow.NET

- As a .NET developer, what I have known about queueing world is only MSMQ because I've rarely had chances to using queues in any projects. I've known RabbitMQ since my last 2 projects and I'm pretty impressed by it's reliability and stability. RabittMQ is very easy to install on Windows or Linux. I wrote a blog post "Install RabbitMQ on SliTaz Linux" which I described step by step to make a RabbitMQ server on such a very tiny SliTaz Linux distro. It's obviously not the first blog post about RabbitMQ in .NET but I would take this as a opportunity to introduce a part of my project: Burrow.NET.

- Burrow.NET is a wrapper library of RabbitMQ.Client for .NET. I started this library from EasyNETQ source code and improve the project to fit our usages. However, I have made this library as generic for any project as possible. Basically, Burrow.NET will abstract away the complexity about connection handling, error handling, logging, and recently from version 1.0.8, I've finished the work around solution for priority queues because RabbitMQ is not natively supporting it. In this post, I'll shortly introduce the very basic features of Burrow.NET and I'll keep other interesting topics for later posts.

I/ Use cases

1/ Let's say you have to build an async web service to serve a huge amount of clients. A normal basic WCF service host on IIS or Window service does not have capability to handle the big amount of requests. So RabbitMQ acts as a middle layer to queue the requests, you can launch as many instance of "processor application" to handle the requests as you need. So this way can easily scale in long term.

2/ You're building a workflow base system that the output of a sub-system will be the input for other system. On other words, your sub-systems need a proper way to asynchronous communicate with each other. Since these sub-systems are different AppDomain so WCF supposes to be a good choice as first sign. However, one like me will not like WCF as an option due to it's configuration complexity. And in this use case, WCF seems not to be a proper solution as we may need scaling for a part or some parts of the system.

- In my personal opinion, whenever you need asynchronous communication between applications, queueing is the best and think about RabbitMQ :D. Well I don't say RabbitMQ is the best. There are alot of mature solutions before RabbitMQ but they came with cost. That's one of the reason why we have the awesome RabbitMQ software and I personaly think RabbitMQ is the best choice, it's fast, it's fashionable, it's free and it works :D.

II/ Burrow.NET

- As the above introduction, this code is a wrapper of .NET RabbitMQ.Client. If you don't want to use wrapper, simply grab RabbitMQ.Client.dll and use it if your need is as simple as publishing a message to a known exchange, queues. However, if you want something consistance like the naming conventions for the whole soultion since you may need alot of queues for the system; or if you expect your application auto reconnect after a connection problem, etc then Burrow.NET is a good candidate.

- I've been using it in my real project which is a big system with different applications comunicate via RabbitMQ channels, I had to make the code as efficient as possible such as establishing only 1 connection for each AppDomain to save the "socket descriptors", open only neccesary RabbitMQ channels to save the resources and consume the messages in the queue as fast as possible to keep the server's memory stay low.

Alright, enough promoting. Here is the main topic:

III/ Example

- Asume that you have to build a web service in user case 1. Your web service will not handle the messages immediately but will put the requests to a queue. There will be 1 or many instances of window services which subscribe to that queue and process them. It's not so complicated, isn't it. Firstly grab the RabbitMQ lib either from NUGET or from GITHUB.

- In other to push messages to RabbitMQ queue, we're gonna need exchange, queue and binding. I really hope you already known about these things. When you publish a message, you actually send it to the RabbitMQ exchange. And depend on the exchange type, also depend on the queue and binding between that queue and the exchange, your messages will be delivered to the proper queue. RabbitMQ has magic stuff to route your messages to the expected target and believe me, you will love that feature. So let's say we will use following exchange and queue:

exchange: RequestExchange
queue : RequestQueue

- The above exchange type will be "direct" and the "RequestQueue" will bind to it with a routing key such as "RequestMessage". That means if you publish a request object to that exchange with a routing key "RequestMessage", it will eventually go to queue "RequestQueue". There are several exchange types, each one will suit different needs and I really recommend you to read about it here: rabbitmq.com/tutorials/tutorial.

- Since we use the above names which are not any convention at all, we'll use ConstantRouteFinder in Burrow.NET.

Here is the code to publish a message:

var routeFinder = new ConstantRouteFinder("RequestExchange", "RequestQueue", "RequestMessage");
var tunnel = RabbitTunnel.Factory.Create();
tunnel.SetSerializer(new JsonSerializer());
tunnel.SetRouteFinder(routeFinder);
tunnel.Publish<RequestMessage>(new RequestMessage());

To subscribe from the above queue, it's very similar

var routeFinder = new ConstantRouteFinder("RequestExchange", "RequestQueue", "RequestMessage");
var tunnel = RabbitTunnel.Factory.Create();
tunnel.SetSerializer(new JsonSerializer());
tunnel.SetRouteFinder(routeFinder);
tunnel.Subscribe<RequestMessage>("RabbitMQDemo", requestMsg => { 
    // Handle the requestMsg here
});

- Or if you want to asynchronous handle the messages in the queue using multi thread:

Global.DefaultConsumerBatchSize = 10; // Set the maximum 10 threads
tunnel.SubscribeAsync<RequestMessage>("RabbitMQDemo", requestMsg => { 
    // Handle the requestMsg here
});

- There something to note here, the code above create RabbitMQ connection using the connection string in your application config. You'll need something like following lines in your app.config:

<connectionStrings>
    <add name="RabbitMQ" connectionString="host=localhost;username=guest;password=guest"/>
</connectionStrings>

- If not, you can use other overloads of RabbitTunnel.Factory to provide the connection string. The "JsonSerializer" is the implementation of ISerializer using Json.NET which is one of the best json serializer for .NET for now. It's currently supporting serializing/deserializing complex object with inheritance hierachy so don't worry about it.

- Burrow.NET has a DefaultRouteFinder which will generate the queue name, exchange name following a default convention which I don't recommend to use. In fact, if you like this library, implement your own IRouteFinder and set that instance to the tunnel like the code above. So I'm saying that If you don't use ConstantRouteFinder as above, the code still work but it will create different exchange and queue following built-in convention. You can checkout the latest source code from GITHUB, there are 2 demo projects which demonstrate everything about the library, and 1 unit test project which reaches 99% test coverage ^^

- That's all you need for publishing and subscribing. Other stuff are already handled by the library and you can change those by implement serveral interfaces then inject to the tunnel using your own TunnelFactory. But that would be other topics, now it's time for me to go to bed :D

Cheers