Skip to content
Related Articles

Related Articles

Improve Article
Save Article
Like Article

How to get Scrapy Output File in XML File?

  • Last Updated : 24 Feb, 2021

Prerequisite: Implementing Web Scraping in Python with Scrapy

Scrapy provides a fast and efficient method to scrape a website. Web Scraping is used to extract the data from websites. In Scrapy we create a spider and then use it to crawl a website. In this article, we are going to extract population by country data from worldometers website.

 Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.  

To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning - Basic Level Course

Let’s implementation with step-wise:

Step 1: Create a Scrapy project



scrapy startproject gfgdemo

We use the above command in the command line to create the project and then change the directory to gfgdemo. After this, we open the project in the editor of our choice (here VS Code).

Step 2: Creating the Spider for Population

scrapy genspider population www.worldometers.info/world-population/population-by-country

A new file named corona.py is added to the spiders directory.

Step 3: Make the following changes to the population.py

Python3




# only keep the base domain
allowed_domains = ['www.worldometers.info'
  
# change http to https

Step 4: Preparing file to scrape the data



First, we will be visiting www.worldometers.info/world-population/population-by-country and disable the javascript as following 

  • Open the inspector tool by pressing Ctrl+Shift+i
  • Then press Ctrl+Shift+p and write javascript and click on the Debugger shown in yellow to disable javascript and then refresh the page.

  • After this, we will select the part to be scraped using the xpath selector.

  • Write the codes for extracting the specific data. We write the following code in the parse method of the spider.

Python3




def parse(self, response):
        countries = response.xpath("//tr")
      
        for country in countries:
            name = country.xpath("(.//td)[2]/a/text()").get()
            population = country.xpath("(.//td)[3]/text()").get()
            yield {
                'name':name,
                'population':population
  
            }

Step 5: Scraping the data

In the command line write the following command to scrape the data for country name and its population.

scrapy crawl population

We get the data as a command-line output as shown above.

To extract the data as an XML file we write the following code in the command line.

scrapy crawl {spider} -o {filename}.xml
Ex: scrapy crawl population -o data.xml

Output:

This will create an XML file in the project directory with the scraped data. The data.xml file here contains the data as

Output:

<?xml version="1.0" encoding="utf-8"?>
<items>
<item><name>None</name><population>None</population></item>
<item><name>China</name><population>1,439,323,776</population></item>
<item><name>India</name><population>1,380,004,385</population></item>
<item><name>United States</name><population>331,002,651</population></item>
<item><name>Indonesia</name><population>273,523,615</population></item>
.
.
.
<item><name>Niue</name><population>1,626</population></item>
<item><name>Tokelau</name><population>1,357</population></item>
<item><name>Holy See</name><population>801</population></item>
</items>




My Personal Notes arrow_drop_up
Recommended Articles
Page :

Start Your Coding Journey Now!