Why do I need middleware like RabbitMQ?
RabbitMQ is a "message broker". Your program gives it messages. Other programs can come along and retrieve those messages and then do something with them. It sits in the middle, so it's often referred to as "middleware".
Why would you want a middleman? Haven't a lifetime of mattress commercials shown that cutting out the middleman is always better?
Not always. Let's build an online store that takes orders and has to tell a warehouse to process the order. Consider these three designs:
- Your website does the work itself.
- Your website passes a message to another program directly.
- Your website passes a message to a broker. Another program picks up messages from the broker.
These are the questions I asked, and below are the conclusions I've come to. If I'm missing anything, please add to the discussion in the comments.
#1 Your website does the work itself
A customer places an order. Since your website is doing all the work itself, it has to: email the customer, charge a credit card, and tell the warehouse to ship the item. What if the warehouse doesn't have the item? Now you have to check another warehouse. While your website is doing this, it has fewer resources available to process other customer's browsing and ordering. So it could become a performance problem as well as a complexity problem. Instead...
#2 Your website passes a message to another program directly.
Let's just tell a warehouse program to do the work, and the website will go about its business. The warehouse program can figure out which warehouse, and send the appropriate information. If it has a web service, we can just push the information. But what if something goes wrong? What if the service is down, or busy, or overloaded? Now we need to build in a retrying mechanism into our website, and we're still managing complexity and putting strain on the website. But what if...
#3 Your website passes a message to a broker. Another program picks up messages from the broker.
Our website just records all the information that the warehouse needs to a message. That message goes on the broker and waits. Once the website dumps the message, it's done, and can go about serving other customers. The warehouse program can ask the broker for messages. It gets a message, does the processing. If it goes well, then the broker can forget the message. If something goes wrong, it's up to the warehouse program to figure out what to do. It could keep retrying, send an email, call a web service, or whatever else you need. If the warehouse program gets overloaded, you can spin up another warehouse program that talks to the same broker. If the warehouse programs crash, the messages will wait on the broker.
It might seem that the middleman's job is very simple, it's very important to ensure that complex operations can be broken down and processed smoothly.
