Skip to content
Related Articles

Related Articles

Scrapy – Settings
  • Last Updated : 03 Mar, 2021

Scrapy is an open-source tool built with Python Framework. It presents us with a strong and robust web crawling framework that can easily extract the info from the online page with the assistance of selectors supported by XPath.

We can define the behavior of Scrapy components with the help of Scrapy settings. Pipelines and setting files are very important for scrapy. It is the core of automating the task. These rules help with inserting data into the database. These files are includes when we start with the base template. The Scrapy settings allow you to customize the behavior of all Scrapy components, including the core, extensions, pipelines, and spiders themselves.

We are often presented with the situation where we need to define multiple crapper projects in that case we can define which individual project with the help of scrapy settings. For this, the environment variable SCRAPY_SETTINGS_MODULE should be used and its value should be in Python path syntax. Hence, with the help of the Scrapy settings, the mechanism for choosing the currently active Scrapy project could be specified.

The infrastructure of the settings provides a worldwide namespace of key-value mappings that the code can use to tug configuration values from. The settings are often populated through different mechanisms, which are described below.

Use these commands to start the scrapy template folder.



scrapy startproject <project_name>

This is the base outline of the scrapy project.

With this article, we would be focusing on the settings.py file. 

The settings.py file looks something like this. We are provided with this as our default settings.

Most commonly used settings and their description is given below:

Important Scrapy Settings

  • BOT_NAME

It is the name of the project. The bot symbolizes the automation that we are doing with the help of the scrapper. It defaults to ‘scrapybot’. Also as seen in the screenshot it is automatically available with your project name when you start the project.

  • USER_AGENT

User-Agent helps us with the identification. It basically tells “who you are” to the servers and network peers. It helps with the identification of the application, OS, vendor, and/or version of the requesting user agent. It defaults to “Scrapy/VERSION (+https://scrapy.org)” while crawling unless explicitly specified. 



The common format for browsers:

User-Agent: <browser>/<version> (<system-info>) <platform> (<platform-details>) <extensions>

For Example:

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
  • ROBOTSTXT_OBEY

A robots.txt file basically tells the crawlers from search engines which pages it could request from the site. ROBOTSTXT_OBEY defaults to “False”. It is mostly kept enabled, so our scrapy will respect the robots.txt policies by the website. 

The image shows the content of the file robots.txt, the policies are written here are managed by the ROBOTSTXT_OBEY setting.

  • CONCURRENT_REQUESTS

It is basically asking the website to open up. It defaults to 16. So basically it is the maximum number of the request that the crawler will perform.

More request increases a load to the server so keeping it as low as 16 or 32 is a good value.

  • CONCURRENT_ITEMS

It means while scrapping the data what a maximum number of concurrent items the scrapy will process in parallel per response. It defaults to 100, which is again a good value.

custom_settings = {
   'CONCURRENT_REQUESTS' = 30,
   'CONCURRENT_ITEMS' = 80,
}
  • CONCURRENT_REQUESTS_PER_DOMAIN

It means while scrapping the data what is the maximum number of existing requests that can be performed concurrently for any single domain value. It defaults to value ‘8’.

  • CONCURRENT_REQUESTS_PER_IP

It means while scrapping the data what is the maximum number of existing requests that can be performed concurrently for any single IP address. It defaults to the value ‘0’.

custom_settings = {
    'CONCURRENT_REQUESTS_PER_DOMAIN' = 8,
    'CONCURRENT_REQUESTS_PER_IP' = 2
}
  • DOWNLOAD_DELAY

It is the delay in the amount of time that the downloader would before again downloading the pages from the website. This again is used to limit the load on the server where the website is hosted. It defaults to 0.

For Example:

DOWNLOAD_DELAY = 0.25    # 250 ms of delay
  • DOWNLOAD_TIMEOUT

It is the time-out time. Tells scrapy to wait for the given amount of time to wait before the downloader times out. It defaults to 180.

  • LOG_ENABLED

It is used to enable or disable the logging for the scrapper. It defaults to “True”.

  • FTP_PASSWORD

Used to set a password for the FTP connections. The value is used only when there is no “ftp_password” in Request meta. It defaults to “guest”.

  • FTP_USER

Used to set a username for the FTP connections. The value is used only when there is no “ftp_user” in Request meta. It defaults to “anonymous”.

  • DEFAULT_ITEM_CLASS

This setting is used to represent items within a scrapy, the values are stored in this class format specified by DEFAULT_ITEM_CLASS. The default format is given by ‘scrapy.item.Item’.

  • DEFAULT_REQUEST_HEADERS

The given setting lists the default header used for HTTP requests made by Scrapy. It is populated within the DefaultHeadersMiddleware.

The default header value is given by:

{
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en',
}
  • REACTOR_THREADPOOL_MAXSIZE

The reactor thread pool could also be set within the scrapy. It binds the max size for the reactor thread pool of the spider. Its default size is 10.

For example, the settings could be applied within the code like the following Python code:

class exampleSpider(scrapy.Spider):
  name = 'example'
  custom_settings = {
      'CONCURRENT_REQUESTS': 25,
      'CONCURRENT_REQUESTS_PER_DOMAIN': 100,
      'DOWNLOAD_DELAY': 0
  }
 
  f = open("example")
  start_urls = [url.strip() for url in f.readlines()]
  f.close()

  def parse(self, response):
      for itemin response.xpath("//div[@class=<class_component>]"):
          urlgem = item.xpath(".//div[@class=<class_component>]/a/@href").extract()
  • AWS_ACCESS_KEY_ID

With this you can set AWS ID within your scrapy, it is used to access Amazon Web Services. It defaults to the “None” value. 

  • AWS_SECRET_ACCESS_KEY

With this you can set AWS Access Key (Password or ID credential) within your scrapy, it is used to access Amazon Web Services. It defaults to the “None” value. 

  • DEPTH_LIMIT

The limiting depth for the spider to crawl a target site. It defaults to 0.

  • DEPTH_PRIORITY

It further manages the priority of the depth to crawl a target site. It also defaults to 0.

This is a basic layout of the selector graph inside the Scrapy. The components could be built inside this Selector Graph. Each component is responsible for scrapping individual items from the site.

  • DEPTH_STATS

With this setting, we can also collect the Depth Stats within the logs of the level crawled. If the setting is enabled then the value of each individual request for each depth is collected in the stats. Its default is “True”.

  • DEPTH_STATS_VERBOSE

Further improves the DEPTH_STATS by enabling the number of requests which are collected in stats for each verbose depth.

By default, it is “False”.

Selector levels can extend up to infinite depth as structured by the webmaster. With the various depth settings, it’s our duty to limit the Selector Graph within our crawler.

  • DNSCACHE_ENABLED

With this setting, we could enable DNS inside a memory cache. By default, it is “True”.

  • DNSCACHE_SIZE

With this setting, we could define the size of the DNS in-memory cache. Its default value is 10000.

  • DNS_TIMEOUT

It is the time-out time for the DNS to process the scrapy query. It defaults to 60.

  • DOWNLOADER

The actual downloader used by the crawler. The default format is given by ‘scrapy.core.downloader.Downloader’.

  • DOWNLOADER_MIDDLEWARES

The dictionary holds downloading middleware and its orders. It is by default empty.

  • EXTENSIONS_BASE

The dictionary with a built-in extension value. It is defaulted by value: { ‘scrapy.extensions.corestats.CoreStats’: 0, }

  • FEED_TEMPDIR

This is a directory that is used to set the custom folder which stores the crawler temporary files.

  • ITEM_PIPELINES

We can define the scrapy dictionary as having pipelines, this represents the pipelines joining each item class. It defaults to the value null.

  • LOG_STDOUT

With this setting, if set to true, all the concurrent process output will appear in the log file. Its default value is False.

Setting up the values

It is advisable to put these values manually inside the settings.py file. Still, there is also an option to modify these values using the command line.

For Example: 

If you want to generate a scrapy log file use the following command.

scrapy crawl myspider -s LOG_FILE=scrapy.log

Conclusion: This is the most important file of the scrapy. Only with this file, you may be able to customize the behaviour of all Scrapy components.

 Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.  

To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning – Basic Level Course

My Personal Notes arrow_drop_up
Recommended Articles
Page :