Python – Asynchronous Task using RabbitMQ
Have you ever faced an issue where you need to perform a task in background that will take up a lot of time to complete? Have you ever wanted to perform a task after a certain interval of time? If your answer was “YES” for any of the above questions, then Python has got you covered. We will be demonstrating how to perform Asynchronous tasks using RabbitMQ.
What exactly Asynchronous means ?
Asynchronous means controlling the timing of operations to be performed by the use of signals sent when the previous operation is completed rather than at regular intervals.
Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.
To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning - Basic Level Course
What RabbitMQ is ?
RabbitMQ is a message-broker software that originally implemented the Advanced Message Queuing Protocol (AMQP) and has since been extended with a plug-in architecture to support Streaming Text Oriented Messaging Protocol (STOMP), MQ Telemetry Transport (MQTT), and other protocols. If that sounded complicated to you then don’t worry, we have got you covered. So, In simple terms, it provides a Queue which performs tasks in the background by running a server of its own.
With all that cleared up, let’s get started with the installation of RabbitMQ and other necessary tools. So, we will be using a python package called Celery for connecting with RabbitMQ. Celery provides an easy way of connecting and sending tasks to the Queue (RabbitMQ). More technically speaking, Celery is a Python Task-Queue system that handle distribution of tasks on workers across threads or network nodes.
You should have python installed on your system. Then you need to install Celery on your system. To do so, simply type the following
pip install celery==4.4.2
Next, install RabbitMQ on your machine. To do so, head over to their official page and download the installer depending on your operating system. You might need o download ErLang along with RabbitMQ. Once installed type the following in your terminal
You should get an output like this
Do not close this terminal. Now your broker should be running. To check that, go to http://localhost:15672/ in your browser and enter the username and password. The default username is guest and the default password is also guest. You should see something like this
Now, let’s get started with the fun part a.k.a. coding.
We will be downloading a YouTube video to our system. We will be using pytube module for downloading it. To install pytube type the following
pip install pytube3
Downloading YouTube video can take up a lot of time. We will now see, how our python file gets executed instantly and the video keeps on downloading in background.
First, we will create a python file called task_queue.py
We need another python file to run this code (you can do it in the same file but it’s a good practice to call the function from another file). Create another file with the name runtask.py
Now, start Celery. To do so, open up a terminal in your working directory and type the following
celery -A task_queue worker --pool=solo -E
If you get any error in this, then either your RabbitMQ server is not running or you are not in your working directory. You should get an output like this
Now we have Celery running. now open up another terminal in the same directory and run the following command
$ python runtask.py https://www.youtube.com/watch?v=vG2PNdI8axo Geeksforgeeks
After you complete the above command, wait for few seconds to allow the download to finish in background. Then go to your D: drive and there you would see a file named Geeksforgeeks.mp4. So, that’s it. Your video is downloaded.
Asynchronous tasks are very essential in real-life cases. You can not have your software or website stuck at a loading page while your code performs tasks. Tasks which take longer time to execute are performed in background without halting the main thread.