The How To Guide for Deploying RabbitMQ on AWS by HashedIn

How to Deploy RabbitMQ on AWS

Technology - 28 Dec 2017
Sripathi Krishnan

RabbitMQ is a messaging broker, and is an excellent choice to maintain task Queues. Here is how you can configure RabbitMQ on AWS in an autoscaling load balanced environment.

Installing RabbitMQ On Ubuntu/Debian

The above script will download the latest version of the RabbitMQ site and install it. Installation instructions can also be referenced from here

Installing RabbitMQ on RPM-based Linux :

Install RabbitMQ-Management Plugin:

RabbitMQ management plugin is a Web UI for administration purposes. You can use it to view the queues, queue elements and a number of consumers. The management plugin is included in the RabbitMQ distribution. To enable it to use the below command.

Create an AMI of this EC2 Instance:

Once you have installed RabbitMQ and configured it, create an AMI out of it. Note the AMI ID, we will use it later in our scripts to automatically scale out or scale in RabbitMQ.

Configuring RabbitMQ Cluster on AWS:

Now that we have our AMI, we will create a cluster on AWS. A cluster provides fault tolerance and load balancing.

Step 1: Create Security Groups RabbitMQ requires several ports to work. Some ports are needed for inter-node communication, others are needed between clients and RabbitMQ, and a third bucket is the HTTP based management interface.

RabbitMQ uses the following ports:

  1. 4369 for epmd
  2. 5672 and 5671
  3. 25672
  4. 15672

If you are deploying this within a VPC (which you should) – then these ports will be open to traffic from other nodes in the VPC. If you are deploying this on an EC2 classic, you will have to create a Security group that allows instances to communicate over these ports.

Step 2: Create a Load Balancer

  1. Create a new classic load balancer
  2. Set the protocol to TCP
  3. Forward port 5672 and 15672 to the instances over TCP/IP
  4. Assign security group that you created in the first step.
  5. In Health Check, ping port 5672
  6. Don’t add any instances yet, because we haven’t launched any

Step 3: Create a Launch Configuration

  1. Choose the AMI that we created earlier
  2. Expand Advanced Details, Copy paste the python script in “UserData” from the section below
  3. Note: Replace the _url value with the DNS Name of the load balancer that was created in Step 1. To get the DNS name, go to load balancer page, select the load balancer that is created, below you can see the DNS name.
  4. Click next until you reach the security group window. Then, Select an existing security group and select the security group that is created in step 1.

Step 4: Create an auto-scaling Group

  1. Check “Create an Auto Scaling group from an existing launch configuration”
  2. Choose the launch configuration that was created from the previous step.
  3. Expand Advanced Details, In Load Balancing, check Receive traffic from Elastic Load Balancer(s). And choose the Load balancer created in step 2.
  4. In the scaling policies, RabbitMQ machines should be scaled based on Memory Utilization.
  5. Remove 1 instances when memory utilization <= 30 for 300 seconds. Add 1 instances when memory utilization >= 70 for 60 seconds
  6. With this configuration, you should see two instances come up in the instances page.

RabbitMQ Launch Configuration Script

This configuration, with go into the Launch Configuration as “User Data”. NOTE: the variable ‘_url’ should be updated with the load balancer URL.

The above code will dynamically add the upcoming instances to the cluster based on the ELB provided in the code as the URL.

What do the call methods mean?

By default, RabbitMQ stays in the reset state. We need to create users and set up the permission for the user as the administrator for future logins. The commands to set up are as follows. The below code block creates User “Admin” with Password “Admin” – change it at your ease.

Once the slaves’ nodes are setup, we need to add a policy to replicate/synchronize the elements of the slave nodes. It can be done using the command in the parent node and it will be applicable across all other nodes in the cluster

You will have to stop the service and fire a command requesting it to join one of the already running rabbitMQ services on other machines. On firing the ‘join_cluster’ request, the cluster between both the machines is automatically created.

To verify, go to the admin page andhttp://<machine-ip>:15672 you will notice 2 nodes, stating that the cluster has been created.

E-book on Digital Business Transformation