Scrapy – Exceptions

Python-based Scrapy is a robust and adaptable web scraping platform. It provides a variety of tools for systematic, effective data extraction from websites. It helps us to automate data extraction from numerous websites.

Scrapy Python

Scrapy describes the spider that browses websites and gathers data in a clear and concise manner. The spider is in charge of accessing the websites, extracting the information, and storing it in a database or a local file. Additionally, complicated websites that employ JavaScript to load data or require authentication can be handled by Scrapy.

In fields like e-commerce, data science, and market research where data extraction from numerous websites is frequently needed, Scrapy is used extensively. It also serves as a framework for teaching web scraping in Python because it is simple to understand.

Exceptions in Scrapy

Errors or extraordinary occurrences that can happen when web scraping are known as exceptions in Scrapy. Invalid data, coding flaws, and network failures are just a few of the causes of these. An exception in Python is a sign that there has been a problem or an unexpected condition that needs to be handled.

Required Module

pip install scrapy

In Scrapy, a few typical instances that can happen are as follows:

DropItem

If you want to remove an item from the pipeline, this exception is triggered. It signals that a certain item should be dropped and not processed any further by the Item Pipeline. To use DropItem, raise the exception and provide an optional reason for dropping the item.

Here, data is missing in the item, hence Dropitem exception will be raised and the item will be dropped for further processing.

Python3

import scrapy
from scrapy.exceptions import DropItem


class MySpider(scrapy.Spider):
    name = "GeeksforGeeks||chsaagar27"

    def parse(self, response):
        item = MyItem()
        if not item.get('data'):
            raise DropItem("Missing data in %s" % item)
        return item

CloseSpider

When you wish to stop the spider from running, you can raise the CloseSpider exception. You may utilize this exception when you have completed your objective or need to halt the spider for any other reason.

Whenever we get our requirements done then we can signal to stop the spider for further processing.

In the example, if the some_condition is met during the spider’s execution, the CloseSpider exception will be raised with the reason “finished”, causing the spider to stop immediately.

Python3

import scrapy
from scrapy.exceptions import CloseSpider

class MySpider(scrapy.Spider):
    name = "GeeksforGeeks||chsaagar27"

    def parse(self, response):
        if some_condition:
            raise CloseSpider("Finished")

In the example, if the some_condition is met during the pipeline processing, the CloseSpider exception will be raised with the reason “Goal reached”, causing the spider to stop immediately.

Python3

import scrapy
from scrapy.exceptions import CloseSpider

class MyPipeline:
    def process_item(self, item, spider):
        if some_condition:
            raise CloseSpider("Finished")
        else:
            return item

IgnoreRequest

This exception is thrown when a spider-generated request is to be disregarded. When you come across an invalid URL or want to exclude particular queries, this can be helpful. It is used when we want to filter out certain requests before they are sent, for example, if the request is a duplicate or if the URL is invalid.

In the example, if the is_valid_url function returns False for a URL, the IgnoreRequest exception will be raised and the corresponding request will not be processed any further.

Python3

import scrapy
from scrapy.exceptions import IgnoreRequest

class MySpider(scrapy.Spider):
    name = "GeeksforGeeks||chsaagar27"

    def start_requests(self):
        for url in urls:
            if not is_valid_url(url):
                raise IgnoreRequest("Invalid URL: %s" % url)
            yield scrapy.Request(url=url, callback=self.parse)

In the example, if the some_condition is met during the middleware processing, the IgnoreRequest exception will be raised and the corresponding request will not be processed any further.

Python3

import scrapy
from scrapy.exceptions import IgnoreRequest

class MyMiddleware:
    def process_request(self, request, spider):
        if some_condition:
            raise IgnoreRequest("Request ignored: %s" % request.url)
        else:
            return None

NotConfigured

When a crucial configuration setting is either absent or ineffective, the NotConfigured exception is thrown. It is used to indicate that the extension or middleware is not properly configured and cannot be used.

In the example, the from_crawler method checks if two required settings (SETTING_1 and SETTING_2) are present in the Scrapy settings. If one or both of the settings are missing, the NotConfigured exception is raised with an appropriate error message.

Python3

import scrapy
from scrapy.exceptions import NotConfigured

class MyExtension:

    def from_crawler(cls, crawler):
        setting1 = crawler.settings.get('SETTING_1')
        setting2 = crawler.settings.get('SETTING_2')

        if not setting1 or not setting2:
            raise NotConfigured("SETTING_1 and \
                        SETTING_2 must be set")

        return cls(setting1, setting2)

NotSupported

This exception is thrown when Scrapy does not support a particular feature or functionality. It is used to indicate that a certain feature or functionality is not supported or implemented. This exception can be useful in situations where you want to indicate to the user or developer that a particular operation or configuration is not allowed or not currently available.

In the example above, if the some_condition is met during the spider’s execution, the NotSupported exception will be raised with an appropriate error message.

Python3

import scrapy
from scrapy.exceptions import NotSupported


class MySpider(scrapy.Spider):
    name = "GeeksforGeeks||chsaagar27"

    def parse(self, response):
        if some_condition:
            raise NotSupported("operation-->not supported")

UsageError

When a command line parameter or option is illegal or absent, this exception is thrown. This exception can be useful in situations where you want to provide feedback to the user or developer about how to correctly use the command or script.

In the example, if the input_file argument is not provided when running the script, the UsageError exception will be raised with an appropriate error message.

Python3

import scrapy
from scrapy.exceptions import UsageError


def my_script(args):

    if not args.input_file:
        raise UsageError("Input file is required")

DeprecationWarning

Use of a deprecated feature or functionality in Scrapy results in the exception ScrapyDeprecationWarning.This warning can be useful in situations where you want to inform the user or developer that they should update their code to use a newer or different approach.

In the example, if my_function() is called, a ScrapyDeprecationWarning warning will be raised with an appropriate message.

Python3

import scrapy
import warnings
from scrapy.exceptions import ScrapyDeprecationWarning

def my_function():
    warnings.warn(
        "my_function() is deprecated, \
                use my_new_function() instead",
        ScrapyDeprecationWarning)

DontCloseSpider 

“DontCloseSpider” is used as that exception which restricts the scrapy to close the spider when it came to the end of the processing of different items and prevents scrapy from being shut down, “DontCloseSpider” is frequently used.

In the example, close_spider() is a function that prevents Scrapy not to close the spider until it is explicitly stopped or encounters an error.

Python3

from scrapy.exceptions import DontCloseSpider
from scrapy.spiders import Spider

class MySpider(Spider):
    name = 'Geeks for Geeks || chsaagar27'

    def parse(self, response):
        # process

    def close_spider(self, spider):
        raise DontCloseSpider

StopDownload

“StopDownload” is recently added in version 2.2 and it is raised in the downloader middleware to restrict the processing of a request and also prevent it from being downloaded. It is best suited for situations when there is a need to filter out specific requests based on particular rules & without actually downloading them.

In the example, requestprocess() is a function that checks for a specific condition is met or not, and if the condition is met(True) then it raises the “StopDownload” exception which stops the processing of the request and prevents it from being downloaded but if the condition is false then the function returns None and let the request to be processed further.

Python3

from scrapy.exceptions import StopDownload


class Middleware:
    def requestprocess(self, request, spider):
        if some_condition:
            raise StopDownload("Request stopped")
        else:
            return None