Skip to content
Related Articles

Related Articles

How to get Scrapy Output File in XML File?
  • Last Updated : 24 Feb, 2021

Prerequisite: Implementing Web Scraping in Python with Scrapy

Scrapy provides a fast and efficient method to scrape a website. Web Scraping is used to extract the data from websites. In Scrapy we create a spider and then use it to crawl a website. In this article, we are going to extract population by country data from worldometers website.

Let’s implementation with step-wise:

Step 1: Create a Scrapy project

scrapy startproject gfgdemo

We use the above command in the command line to create the project and then change the directory to gfgdemo. After this, we open the project in the editor of our choice (here VS Code).

Step 2: Creating the Spider for Population

scrapy genspider population

A new file named is added to the spiders directory.

Step 3: Make the following changes to the


# only keep the base domain
allowed_domains = [''
# change http to https

Step 4: Preparing file to scrape the data

First, we will be visiting and disable the javascript as following 

  • Open the inspector tool by pressing Ctrl+Shift+i
  • Then press Ctrl+Shift+p and write javascript and click on the Debugger shown in yellow to disable javascript and then refresh the page.

  • After this, we will select the part to be scraped using the xpath selector.

  • Write the codes for extracting the specific data. We write the following code in the parse method of the spider.


def parse(self, response):
        countries = response.xpath("//tr")
        for country in countries:
            name = country.xpath("(.//td)[2]/a/text()").get()
            population = country.xpath("(.//td)[3]/text()").get()
            yield {

Step 5: Scraping the data

In the command line write the following command to scrape the data for country name and its population.

scrapy crawl population

We get the data as a command-line output as shown above.

To extract the data as an XML file we write the following code in the command line.

scrapy crawl {spider} -o {filename}.xml
Ex: scrapy crawl population -o data.xml


This will create an XML file in the project directory with the scraped data. The data.xml file here contains the data as


<?xml version="1.0" encoding="utf-8"?>
<item><name>United States</name><population>331,002,651</population></item>
<item><name>Holy See</name><population>801</population></item>

Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.

To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course.

My Personal Notes arrow_drop_up
Recommended Articles
Page :