Scrapy – Spiders

Scrapy is a free and open-source web-crawling framework which is written purely in python. Thus, scrapy can be installed and imported like any other python package. The name of the package is self-explanatory. It is derived from the word ‘scraping’ which literally means extracting desired substance out of anything physically using a sharp tool. Scraping in web technology refers to an automated process in which bots called spiders are used to extract useful content and data from websites. 

Python scrapy package

This package is maintained and managed by an organization called Scrapinghub Ltd. This package comes with handy command line tools which makes project management an easy task.

The major use cases of this package are as follows:-

• The primary objective of this package is web scraping.
• It is used for building regular web crawlers and for extracting data using APIs. 
• It is a complete package that performs extraction as well as storage of collected data in databases. 

Features of scrapy

• Scrapy provides multiple ways to scrape the same website.
• It is faster when compared to other scraping tools as it can scrape multiple webpages and URLs at the same time. 
• Scrapy is capable of scraping websites concurrently.
• Item pipeline is an important feature of scrapy that enables the user to replace the value of scraped data. 

Web spider

Web spiders are called by technocrats using different names. The other names of web spider are web crawler, automatic indexer, crawler or simply spider. So don’t get confused with all these fancy names since they all refer to the same thing. A web spider is actually a bot that is programmed for crawling websites. 

The primary duty of a web spider is to generate indices for websites and these indices can be accessed by other software. For instance, the indices generated by a spider could be used by another party to assign ranks for websites. 

Steps to create a basic web spider 

To understand the basics of a  web spider in programming terminology, let’s build our own spider in python using scrapy. 

Step 1: Create a virtual environment

When you install scrapy in your system, it may conflict with other processes that are running. There are chances of getting errors and possible crashes while trying to use scrapy within a system. Although this step is not mandatory, I strongly recommend you to install scrapy in a virtual environment. 

A virtual environment provides an isolated container for your scrapy project which will not interfere with your actual system, thus preventing possible conflicts. I also suggest you to use pycharm as it creates a virtual environment for you automatically every time you create a new project. 

Step 2: Install python scrapy package

We can easily install the package from the terminal using Preferred Installer Program (PIP). Here I’m installing the package in the project that I have created- ‘first_spider’ Execute the following command in the command line interface you use. 

(venv) PS C:\Users\Kresh\PycharmProjects\first_spider> pip install scrapy

This will install the package along with all of it’s dependencies such as w3lib and lxml. 

Note: The installed package could only be accessed from the virtual environment in which you have installed the package. 

Step 3: Start a new scrapy project 

Scrapy has a resourceful command line utility. To know about various commands available in scrapy, type the following command in the terminal

(venv) PS C:\Users\Kresh\PycharmProjects\first_spider> scrapy

To know about the options available for  a specific command, type the following command.

(venv) PS C:\Users\Kresh\PycharmProjects\first_spider> scrapy <command_name> -h 

Enter the below command to start a scrapy project using the scrapy command-line utility

(venv) PS C:\Users\Kresh\PycharmProjects\first_spider> scrapy startproject <project_name>

Note: commands inside angular brackets ‘<>’ need to be replaced by your own data. 

Your project will now be created with a built-in tree structure of directories. 

Step 4: Create the spider

In the tree structure that has been created automatically, you can find  a directory called ‘spiders’. You should create all your spiders inside this directory only. 

Right click the directory - 'spider' --> new --> python file

Follow the above steps to create a new spider. It can be given any name of your choice but it must be a python file (.py). In our case, the name of the spider is ‘gfg_spider1’

spider created with the name – ‘gfg_spider1.py’

Now  you can start constructing the code. 

Now that you have learned where to type your source code for a spider. It is now time to create some spiders and see what they can do. The below examples will help you in getting a foundational knowledge about spiders.

Example1: Extracting HTML content of a webpage

Python3

# Spider dedicated to GeekForGeeks

# Importing necessary packages
import scrapy

# Creating the spider class


class GFGSpider (scrapy.Spider):
    # Assigning a name to the created spider
    name = "gfg"

    # Defining start_requests() function
    def start_requests(self):
        url_list = [""]     # Enter your target URL here
        for url in url_list:
            yield scrapy.Request(url=url, callback=self.parse)

    # Defining parse() function
    def parse(self, response):
        page = response.url.split("/")[-2]
        file_name = "gfg-%s.html" % page    # Name of the output file
        with open(file_name, 'wb') as file:
            file.write(response.body)  # Writing the file
        self.log("file saved %s" % file_name)

Save the file and run the program by entering the below command in the terminal.

(venv) PS C:\Users\Kresh\PycharmProjects\first_spider> scrapy crawl gfg

Note:-

• Replace gfg with the name of your spider , i.e, the value assigned to ‘name’ inside the spider class.
• In the source code, fill the empty ‘url_list’ with the target URL before running the spider.

Explanation of Code

1. In context of python programming language, spider is just a class that has special methods associated with it to crawl and scrape webpages. So lets start with creating a class that inherits the base Spider class.
2. ‘name’ is an attribute that defines the name of the spider. It must be unique.
3. start_request() is a method that is called by scrapy when a spider has opened. It is called only once. It is used to initiate the process of scraping and can be used as a generator. 
4. URLs are stored in a list and are scraped one by one using a ‘for’ loop.
5. The ‘yield’ keyword is an inbuilt feature of Spider class that saves the data acquired after the completion of a request.
6. parse() method is scrapy’s default callback method. Thus we don’t need to use a callback explicitly to call this method. It is used for processing the received response. The response received is given as a parameter to this method.  
7. In the parse() method, the body of  response is written into the html file created, using file-handling methods. 

Output:

The output will get saved as a HTML file which can be seen in our scrapy project.

Example 2: Extraction of image files from webpages

Python3

# import section
import scrapy

# spider inheriting Spider base class


class GFGImageSpider (scrapy. Spider):
    name = "gfg_img"
    start_urls = ["https://practice.geeksforgeeks.org/"]

    def parse(self, response):
        # using xpath selector
        links = response.xpath("//img/@src")
        # to store html content in the form of string
        html = ""

        for link in links:
            # converting link to string
            url = link.get()
            # checking for images
            if any(extension in url for extension in [".jpg",
                                                      ".gif",
                                                      ".png"]):
                html += """<a href="{url}"
                target="_blank">
                <img src="{url}" height="33%"
                width="33%" />
                <a/>""". format(url=url)
                # creating a file to write html content
                with open("gfg_images.html", "a") as page:
                    page.write(html)
                    page.close()

Explanation of code

• Import scrapy.
• Create the regular spider template. The spider class should inherit the Spider base class. Also define a parse() method. Provide a list of start URLs and assign a unique name to your spider using the keyword “name”.
• The parse method is going to yield the desired output. So we are using a selector tool based on xpath that traverses the tree structure of elements in the webpage. Images are stored in the ‘src’ attribute of <img> tag in HTML documents. Hence we are providing the path of <img> tag and specifying the attribute we are looking for using ‘@’ symbol.
• While parsing we are going to dynamically add the selected content to a variable which can then be written to a file. For this we are declaring an empty string called ‘html’.
• Initiate a ‘for’ loop to iterate through all the retrieved links.
• Convert the retrieved links to strings using the get() method.
• In the ‘if’ condition, look for string that matches common image file extensions such as .jpg, .gif, .png.
• Build the structure of the HTML file. We can do this by appending multi line strings that has <a> tags containing image within them.
• Make sure to give the values of all attributes within curly brackets to enable string formatting. It allows us to build our file dynamically.
• You can provide the dimensions of the image using ‘width’ and ‘height’ attributes.
• Now use the common file handling method open() to create our HTML document and write the dynamically written HTML content stored in ‘html’ variable.
• Close the file after writing it.
• After crawling the webpage, the spider will generate the expected HTML file which will contain all the extracted images.

Output

Example 3:  Selective crawling

In the previous examples we have used inherited the ‘Spider’ class to create a spider. This class is also called BasicSpider or GenericSpider because it is simple and doesn’t have many functionalities. In this example, we are going to use the CrawlSpider class which allows us to define the behavior of  spider using rules.

Python3

# import section
import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor


class GFGSpidey(CrawlSpider):
    name = "gfg_spidey"
    # list of URLs to scrape
    allowed_domains = ["geeksforgeeks.org"]
    start_urls = ["https://www.geeksforgeeks.org/category/c-arrays/"]
    # creation of rules
    rules = (
        Rule(LinkExtractor(allow="c-arrays"), callback="parse_item"),
    )
    # method to parse

    def parse_item(self, response):
        # saving items via yield dictionary
        yield{
            "Title of article": response.css("div.head a::text").get(),
            "Tag present": response.css("div.item a::text").get()
        }

Explanation of code

• Import the necessary classes – CrawlSpider, Rule and LinkExtractor.
• Create the usual template of a spider i.e, the spider class with a list of start URLs and allowed domain names and also a parse() method. Note that the parse() method should have a name other than ‘parse’ to avoid overriding of the default parse() method. So we have named it as ‘parse_item()’. Also, inherit the CrawlSpider instead of the Spider base class.
• Now define  the rule using the Rule object. The attributes of LinkExtractor includes ‘allow’ and ‘deny’. These attributes are used to select or reject a location respectively. In this example we are going to scrape URL with the string ‘c-arrays’ in it. The ‘callback’ attribute is to call our parse_item() function when the defined URL is encountered.
• We need to store and save the required output. Scrapy comes with a predefined tool called ‘yield’ which can be easily used instead of normal file handling methods to fetch and display the required content. We can bring this tool into action using the keyword ‘yield’. Yield accepts data only in the form of a Request or BaseItem or Dict or None. We are using a dictionary here.
• Let’s specify the contents that we require inside the dictionary. One of the easiest way to select content is based on CSS, using the response.css() method. You can specify the hierarchy of elements and the value in this method to fetch the content. 
• you can identify the CSS selectors used in the required content by inspecting the webpage. Right click anywhere in the webpage and select ‘inspect’ or use the shortcut ‘ctrl+shift+c‘ in windows to open developer tools from where you can inspect the source code of the web page.

Inspection of webpage

• In our case, I’m inspecting the tag of the article which is under a division with a class value of ‘item’ and I want the text enclosed by anchor tags. In response.css(), a class name follows a dot ‘.’ and a double colon ‘::’ is used to separate the tag name and the value to fetch. In this example the tag name is ‘a’ for anchor and value is ‘text’ as we need the enclosed text content only.
• In the same way, inspect the title of article and use the location in another response.css() method.
• The title and tag of each article from URL that contains ‘c-arrays’ is successfully extracted.

Note: Although web scraping is absolutely legal, most of the websites cannot be scraped since they prevent bot activities for security reasons. After all, web spider is indeed a bot!

Finding websites to scrape

Type ‘robots.txt’ at the end of the URL of the website that you need to scrape. The robots.txt is a file that gives you a list of bots that are not allowed in the website. If spiders are not present in the list of disallowed bots, you might scrape that website.

robots.txt

Applications of web spiders

• Web spiders can be used to compare the price of the same product in multiple e-commerce platforms. It saves you a lot of time and money.
• Spiders can extract specific data from a website using filters . For example, extraction of email IDs. reference:- https://www.geeksforgeeks.org/email-id-extractor-project-from-sites-in-scrapy-python/amp/
•  They allow us to get a better understanding about the structure of websites. It provides detailed information about various tools and languages used in them. Thus could be used as a pro web developer tool.
• They are used extensively for data mining and data analysis especially to study inbound and outbound links.
• They are also used in information hubs such as news portals.