How to get Scrapy Output File in XML File?

Prerequisite: Implementing Web Scraping in Python with Scrapy

Scrapy provides a fast and efficient method to scrape a website. Web Scraping is used to extract the data from websites. In Scrapy we create a spider and then use it to crawl a website. In this article, we are going to extract population by country data from worldometers website.

Let’s implementation with step-wise:

Step 1: Create a Scrapy project

scrapy startproject gfgdemo

We use the above command in the command line to create the project and then change the directory to gfgdemo. After this, we open the project in the editor of our choice (here VS Code).

Step 2: Creating the Spider for Population

scrapy genspider population www.worldometers.info/world-population/population-by-country

A new file named corona.py is added to the spiders directory.

Step 3: Make the following changes to the population.py

Python3

# only keep the base domain
allowed_domains = ['www.worldometers.info']

# change http to https
start_urls = ['https://www.worldometers.info/world-population/population-by-country/']

Step 4: Preparing file to scrape the data

First, we will be visiting www.worldometers.info/world-population/population-by-country and disable the javascript as following 

• Open the inspector tool by pressing Ctrl+Shift+i
• Then press Ctrl+Shift+p and write javascript and click on the Debugger shown in yellow to disable javascript and then refresh the page.

• After this, we will select the part to be scraped using the xpath selector.

• Write the codes for extracting the specific data. We write the following code in the parse method of the spider.

Python3

def parse(self, response):
        countries = response.xpath("//tr")

        for country in countries:
            name = country.xpath("(.//td)[2]/a/text()").get()
            population = country.xpath("(.//td)[3]/text()").get()
            yield {
                'name':name,
                'population':population

            }

Step 5: Scraping the data

In the command line write the following command to scrape the data for country name and its population.

scrapy crawl population

We get the data as a command-line output as shown above.

To extract the data as an XML file we write the following code in the command line.

scrapy crawl {spider} -o {filename}.xml
Ex: scrapy crawl population -o data.xml

Output:

This will create an XML file in the project directory with the scraped data. The data.xml file here contains the data as

Output:

<?xml version="1.0" encoding="utf-8"?>
<items>
<item><name>None</name><population>None</population></item>
<item><name>China</name><population>1,439,323,776</population></item>
<item><name>India</name><population>1,380,004,385</population></item>
<item><name>United States</name><population>331,002,651</population></item>
<item><name>Indonesia</name><population>273,523,615</population></item>
.
.
.
<item><name>Niue</name><population>1,626</population></item>
<item><name>Tokelau</name><population>1,357</population></item>
<item><name>Holy See</name><population>801</population></item>
</items>