Logging in Scrapy

Scrapy is a fast high-level web crawling and scraping framework written in Python used to crawl websites and extract structured data from their pages. It can be used for many purposes, from data mining to monitoring and automated testing.

As developers, we spend most of our time debugging than writing new code. Logging is one of the techniques that is used to make debugging easier. It refers to keeping track of the log of events, including errors, problems, etc., that arise during the code’s runtime.

Logging in Scrapy:

Initially, Scrapy provided the logging feature through the scrapy.log module.

But it is deprecated now and no longer supported.

Instead, python’s built-in logging module can be used along with Scrapy to log its events. 

Python’s built-in logging has defined 5 different levels to indicate the severity of a given log message as listed below in Decreasing order of severity:

• Level 5: logging.CRITICAL – for critical errors [Highest severity]

Python3

import logging
logging.critical("Scrapy Log to display Critical messages")

• Level 4: logging.ERROR – for regular errors

Python3

import logging
logging.error("Scrapy Log to display Error messages")

• Level 3: logging.WARNING – for warning messages

Python3

import logging
logging.warning("Scrapy Log to display Warning messages")

• Level 2: logging.INFO – for informational messages

Python3

import logging
logging.info("Scrapy Log to display Info messages")

• Level 1: logging.DEBUG – for debugging messages [Lowest severity]

Python3

import logging
logging.debug("Scrapy log to display Debugging messages")

Scrapy Spider Logs:

Scrapy supports a Logger inside each Spider instance. It can be accessed and used as shown below:

A step-by-step method for logging in spiders:

1. Installation of packages – run the following command from the terminal 

pip install scrapy

2.  Create a Scrapy project – run the following command from the terminal

scrapy startproject scrapy_log
cd scrapy_log
scrapy genspider log http://books.toscrape.com/

Here,

• Project Name:  “scrapy_log”
• Spider Name: “log”
• Domain to be Scraped: “http://books.toscrape.com/” 

4. Define the Parse function –  Add the following code to “scrapy_log\spiders\log.py”

• To create a logger with the name of the spider: (i.e. “log”)

Python3

import scrapy

class LogSpider(scrapy.Spider):
    name = 'log'
    allowed_domains = ['books.toscrape.com']
    start_urls = ['http://books.toscrape.com/']

    def parse(self, response):
        self.logger.info('Parse function called on %s', response.url)

Spider Named Logger

• To create a user-defined custom-named logger: (i.e. “GFG_logger”)

Python3

import scrapy
import logging

logger = logging.getLogger('GFG_logger')

class LogSpider(scrapy.Spider):
    name = 'log'
    allowed_domains = ['books.toscrape.com']
    start_urls = ['http://books.toscrape.com/']

    def parse(self, response):
        logger.info('Parse function called on %s', response.url)

Custom Named Logger

• To create a custom Logger Format:

The Logging basic configuration is defined in the below code as follows:

• level – Defines till which level of messages should be logged starting from level 1
• format – Defines the general format of the log messages – (“[DateTime] {LoggerName}  LevelName:    Message”)
• datefmt – Defines the format of the Timestamp that is displayed

Python3

import scrapy
import logging

logging.basicConfig(level=logging.CRITICAL,
                    format='[%(asctime)s] {%(name)s} %(levelname)s:  %(message)s',
                    datefmt='%y-%m-%d %H:%M:%S')
logger = logging.getLogger('GFG_logger')

class LogSpider(scrapy.Spider):
    name = 'log'
    allowed_domains = ['books.toscrape.com']
    start_urls = ['http://books.toscrape.com/']

    def parse(self, response):
        logger.info('Parse function called on %s', response.url)

Custom Format Logger

• To Export the logs to a Log File:

The logs can be saved to a Log File as shown in the below code where it saves the logs to a file named (“saved_logs.log”)

Python3

import scrapy
import logging

logging.basicConfig(level=logging.CRITICAL,
                    format='[%(asctime)s] {%(name)s} %(levelname)s:  %(message)s',
                    datefmt='%y-%m-%d %H:%M:%S',
                    filename="saved_logs.log")
logger = logging.getLogger('GFG_logger')

class LogSpider(scrapy.Spider):
    name = 'log'
    allowed_domains = ['books.toscrape.com']
    start_urls = ['http://books.toscrape.com/']

    def parse(self, response):
        logger.info('Parse function called on %s', response.url)

saved_logs.log file

5. Run the spider using either of the following commands:

scrapy crawl log

The above command lists all the logs.

scrapy crawl log -L INFO

Here, “-L” is used to specify the Log level that needs to be listed (i.e. INFO/DEBUG/CRITICAL/WARN/ERROR)