Python – Asynchronous Task using RabbitMQ

Have you ever faced an issue where you need to perform a task in background that will take up a lot of time to complete? Have you ever wanted to perform a task after a certain interval of time? If your answer was “YES” for any of the above questions, then Python has got you covered. We will be demonstrating how to perform Asynchronous tasks using RabbitMQ.

What exactly Asynchronous means ?
Asynchronous means controlling the timing of operations to be performed by the use of signals sent when the previous operation is completed rather than at regular intervals.

What RabbitMQ is ?
RabbitMQ is a message-broker software that originally implemented the Advanced Message Queuing Protocol (AMQP) and has since been extended with a plug-in architecture to support Streaming Text Oriented Messaging Protocol (STOMP), MQ Telemetry Transport (MQTT), and other protocols. If that sounded complicated to you then don’t worry, we have got you covered. So, In simple terms, it provides a Queue which performs tasks in the background by running a server of its own.

With all that cleared up, let’s get started with the installation of RabbitMQ and other necessary tools. So, we will be using a python package called Celery for connecting with RabbitMQ. Celery provides an easy way of connecting and sending tasks to the Queue (RabbitMQ). More technically speaking, Celery is a Python Task-Queue system that handle distribution of tasks on workers across threads or network nodes.

You should have python installed on your system. Then you need to install Celery on your system. To do so, simply type the following

pip install celery==4.4.2

Next, install RabbitMQ on your machine. To do so, head over to their official page and download the installer depending on your operating system. You might need o download ErLang along with RabbitMQ. Once installed type the following in your terminal

rabbitmq-server restart

You should get an output like this

Do not close this terminal. Now your broker should be running. To check that, go to http://localhost:15672/ in your browser and enter the username and password. The default username is guest and the default password is also guest. You should see something like this

Now, let’s get started with the fun part a.k.a. coding.

We will be downloading a YouTube video to our system. We will be using pytube module for downloading it. To install pytube type the following

pip install pytube3

Downloading YouTube video can take up a lot of time. We will now see, how our python file gets executed instantly and the video keeps on downloading in background.

First, we will create a python file called task_queue.py

from celery import Celery
import sys

from pytube import YouTube

# Where the downloaded files will be stored
BASEDIR ="D:\\"

# Create the app and set the broker location (RabbitMQ)
app = Celery('downloader',
            backend ='rpc://',
            broker ='pyamqp://guest@localhost//')

@app.task
def download(url, filename):
    """
    Download a page and save it to the BASEDIR directory
      url: the url to download
      filename: the filename used to save the url in BASEDIR
    """
    try:
        # object creation using YouTube which
        # was imported in the beginning
        yt = YouTube(url)
    except:
        print("Connection Error") # to handle exception

    # filters out all the files with "mp4" extension
    yt.streams\
        .filter(progressive = True, file_extension ='mp4')\
        .order_by('resolution')[-1]\
        .download(output_path = BASEDIR, filename = filename)
        # downloading the video

    print('Task Completed !')

We need another python file to run this code (you can do it in the same file but it’s a good practice to call the function from another file). Create another file with the name runtask.py

import sys
from task_queue import download

# gets the first command line argument
link = sys.argv[1]

# gets the second command line argument
filename = sys.argv[2]

# calling the download function
download.delay(link, filename)

Now, start Celery. To do so, open up a terminal in your working directory and type the following

celery -A task_queue worker --pool=solo -E

If you get any error in this, then either your RabbitMQ server is not running or you are not in your working directory. You should get an output like this

Now we have Celery running. now open up another terminal in the same directory and run the following command

$ python runtask.py https://www.youtube.com/watch?v=vG2PNdI8axo Geeksforgeeks

After you complete the above command, wait for few seconds to allow the download to finish in background. Then go to your D: drive and there you would see a file named Geeksforgeeks.mp4. So, that’s it. Your video is downloaded.

Asynchronous tasks are very essential in real-life cases. You can not have your software or website stuck at a loading page while your code performs tasks. Tasks which take longer time to execute are performed in background without halting the main thread.