Scrapy – Item Loaders

In this article, we are going to discuss Item Loaders in Scrapy.

Scrapy is used for extracting data, using spiders, that crawl through the website. The obtained data can also be processed, in the form, of Scrapy Items. The Item Loaders play a significant role, in parsing the data, before populating the Item fields.  In this article, we will learn about Item Loaders.

Installing Scrapy:

Scrapy, requires a Python version, of 3.6 and above. Install it, using the pip  command, at the terminal as:

pip install Scrapy  

This command will install the Scrapy library, in your environment. Now, we can create a Scrapy project, to write the Python Spider code.

Create a Scrapy Spider Project

Scrapy comes with an efficient command-line tool, called the Scrapy tool. The commands have a different set of arguments, based on their purpose. To write the Spider code, we begin by creating, a Scrapy project. Use the following, ‘startproject’ command, at the terminal –  

scrapy startproject gfg_itemloaders

This command will create a folder, called ‘gfg_itemloaders’. Now, change the directory, to the same folder, as shown below –

Use ‘startproject’ command to create Scrapy project

The folder structure, of the scrapy project, is as shown below:

The folder structure of Scrapy project

It has a scrapy.cfg file, which, is the project configuration file. The folder, containing this file, is called as the root directory. The directory, also has items.py, middleware.py, and other settings files, as shown below –

The folder structure of Scrapy project

The spider file, for crawling, will be created inside the ‘spiders’ folder. We will mention, our Scrapy items, and, related loader logic, in the items.py file. Keep the contents of the file, as it is, for now. Using ‘genspider’ command, create a spider code file. 

scrapy genspider gfg_loadbookdata “books.toscrape.com/catalogue/category/books/womens-fiction_9”

The command, at the terminal, is as shown below –

Use ‘genspider’ command to create spider file

Data  Extraction Using Scrapy Items

We will scrape the Book Title, and, Book Price, from the Women’s fiction webpage. Scrapy, allows the use of selectors, to write the extraction code. They can be written, using CSS or XPath expressions, which traverse the entire HTML page, to get our desired data. The main objective, of scraping, is to get structured data, from unstructured sources. Usually, Scrapy spiders will yield data, in Python dictionary objects. The approach is beneficial, with a small amount of data. But, as your data increases, the complexity increases. Also, it may be desired, to process the data, before we store the content, in any file format. This is where, the Scrapy Items, come in handy. They allow the data,  to be processed, using Item Loaders. Let us write, Scrapy Item for Book Title and Price, and, the XPath expressions, for the same.

‘items.py’ file, mention the attributes, we need to scrape.

We define them as follows:

Python3

# Define here the models for your scraped item
import scrapy

# Item class name for the book title and price
class GfgItemloadersItem(scrapy.Item):

    # Scrape Book price
    price = scrapy.Field()

    # Scrape Book Title
    title = scrapy.Field()

• Please note that Field() allows, a way to define all field metadata, in one location. It does not provide, any extra attributes.
• XPath expressions, allow us to traverse the webpage, and, extract the data. Right-click, on one of the books, and, select the ‘Inspect’ option. This should show its HTML attributes, in the browser. All the books on the webpage, are contained, in the same <article> HTML tag, having class attribute, as ‘product_pod’.  It can be seen as below –

All books belong to the same ‘class’ attribute ‘product_pod’

• Hence, we can iterate through, the <article> tag class attribute, to extract all Book titles and Price, on the webpage. The XPath expression , for the same, will be books =response.xpath(‘//*[@class=”product_pod”]’). This should return, all the book HTML tags, belonging to the class attribute as “product_pod”. The ‘*’ operator indicates, all tags, belonging to the class ‘product_pod’. Hence, we can now have a loop, that navigates to each and every Book, on the page.
• Inside the loop, we need to get the Book Title. Hence, right-click on the title and choose ‘Inspect’. It is included, in <a> tag, inside header <h3> tag. We will fetch the “title” attribute of the <a> tag. The XPath expression, for the same, would be, books.xpath(‘.//h3/a/@title’).extract(). The dot operator indicates, we will be using the ‘books’ object now, to extract data from it. This syntax will traverse through the header, and then, <a> tag, to get the title of the book.
• Similarly, to get the Price of the book, right click and say Inspect on it, to get its HTML attributes. All the price elements, belong to the <div> tag, having class attribute as “product_price”. The actual price is mentioned, inside the paragraph tag, present, inside the <div> element. Hence, the XPath expression, to get the actual text of Price, would be books.xpath(‘.//*[@class=”product_price”]/p/text()’).extract_first(). The extract_first() method, returns, the first price value.

We will create, an object of the above, Item class, in the spider, and, yield the same. The spider code file will look as follows:

Python3

# Import Scrapy library
import scrapy

# Import Item class
from ..items import GfgItemloadersItem

# Spider class name
class GfgLoadbookdataSpider(scrapy.Spider):

    # Name of the spider
    name = 'gfg_loadbookdata'

    # The domain to be scraped
    allowed_domains = [
        'books.toscrape.com/catalogue/category/books/womens-fiction_9']

    # The URL to be scraped
    start_urls = [
        'http://books.toscrape.com/catalogue/category/books/womens-fiction_9/']

    # Default parse callback method
    def parse(self, response):

        # Create an object of Item class
        item = GfgItemloadersItem()

        # loop through all books
        for books in response.xpath('//*[@class="product_pod"]'):

            # XPath expression for the book price
            price = books.xpath(
                './/*[@class="product_price"]/p/text()').extract_first()

            # place price value in item key
            item['price'] = price

            # XPath expression for the book title
            title = books.xpath('.//h3/a/text()').extract()

            # place title value in item key
            item['title'] = title

            # yield the item
            yield item

• When we execute, the above code, using scrapy “crawl” command, using the syntax as, scrapy crawl spider_name, at the terminal as –

scrapy crawl gfg_loadbookdata -o not_parsed_data.json

The data is exported, in the “not_parsed_data.json” file, which can be seen as below:

The items yielded when data is not parsed

Now, suppose we want to process, the scraped data, before yielding and storing them, in any file format, then we can use Item Loaders.

Introduction to Item Loaders

Item loaders, allow a smoother way, to manage scraped data. Many times, we may need to process, the data we scrape. This processing can be:

• Refining or editing the text present.
• Replacing any characters present, with another, or, replace missing data, with proper characters.
• Strip undesired characters.
• Clean whitespace characters.

In this article, we will do the following processing –

• Remove, the ‘£’ (pound) currency, from the Book Price.
• Replace, the ‘&’ sign wherever present in the Book title, with ‘AND’.

How do Item Loaders work?

So far we know, Item Loaders are used to parse, the data, before Item fields are populated. Let us understand, how Item Loaders work –

• Item loaders, help in populating, the scraped data, into Scrapy Items. The Items are fields, defined in the ‘items.py’ file.
• An Item Loader will have one input processor, and, one output processor, defined for each Item field.
• We know, Scrapy makes use of Selectors, which are XPath or CSS expressions, to navigate to the desired HTML tag.
• The Item loader, uses, its add_xpath() or add_css() methods, to fetch the data desired.
• The Input processors, then act on this data. We can mention, our custom functions, as parameters, to input processors, to parse, the data as we want.
• The result, of the input processor, is stored in the ItemLoader.
• Once, all the data is received, and, parsed, according to input_processor, the loader will call, its load_item() method, to populate the Item object.
• During this process, the output processor is called, and, it acts on that intermediate data.
• The result of the output processor is assigned to the Item object.
• This is how, parsed Item objects, are yielded.

Built-in processors:

Now, let us understand, the built-in processors, and, methods that we will use, in Item Loaders, implementation. Scrapy has six built-in processors. Let us know them –

Identity(): This is the default, and, simplest processor. It never changes any value. It can be used, as an input, as well as, output processor. This means, when no other processor, is mentioned, this acts, and, returns the values unchanged.

Python3

# Import the processor
from itemloaders.processors import Identity

# Create object of Identity processor
proc = Identity()

# Assign values and print result
print(proc(['star','moon','galaxy']))

Output:

['star','moon','galaxy']

TakeFirst(): This returns, the first non-null, or, non-empty value, from the data received. It is usually, used as an output processor.

Python3

# import the processor module
from itemloaders.processors import TakeFirst

# Create object of TakeFirst processor
proc = TakeFirst()

# assign values and print the result
print(proc(['', 'star','moon','galaxy']))

Output:

'star'

Compose(): This takes data, and, passes it to the function, present in the argument. If more than one function, is present in the argument, then the result of the previous, is passed to the next. This continues, till the last function, is executed, and, the output is received.

Python3

# Import the processor module
from itemloaders.processors import Compose

# Create an object of Compose processor and pass values
proc = Compose(lambda v: v[0], str.upper)

# Assign values and print result
print(proc(['hi', 'there']))

Output:

HI

MapCompose(): This processor, works similarly to Compose. It can have, more than one function, in the argument. Here, the input values are iterated, and, the first function, is applied to all of them, resulting in a new iterable. This new iterable is now passed to the second function, in argument, and so on. This is mainly used, as an input processor. 

Python3

# Import MapCompose processor
from itemloaders.processors import MapCompose

# custom function to filter star
def filter_star(x):

    # return None if 'star' is present
    return None if x == 'star' else x

# Assign the functions to MapCompose
proc = MapCompose(filter_star, str.upper)

# pass arguments and print result
print(proc(['twinkle', 'little', 'star','wonder', 'they']))

Output:

['TWINKLE', 'LITTLE', 'WONDER', 'THEY']

Join(): This processor, returns the values joined together. To put an expression, between each item, one can use a separator, the default one is ‘u’. In the example below, we have used <a> as a separator:

Python3

# Import required processors
from itemloaders.processors import Join

# Put separator <br> while creating Join() object
proc = Join('<a>')

# pass the values and print result
print(proc(['Sky', 'Moon','Stars']))

Output:

'Sky<a>Moon<a>Stars'

SelectJmes(): This processor, using the JSON path given, queries the value and returns the output.

Python3

# Import the class
from itemloaders.processors import SelectJmes

# prepare object of SelectJmes
proc = SelectJmes("hello")

# Print the output of json path
print(proc({'hello': 'scrapy'}))

Output:

scrapy

In this example, we have used TakeFirst() and MapCompose() processors. The processors, act on the scraped data, when Item loader functions, like add_xpath() and others, are executed. The most commonly used, loader functions are –

• add_xpath() – This method, takes the item field, and, corresponding XPath expression for it. It mainly accepts parameters as –
  • field_name – The item field name, defined in the ‘items.py’ class.
  • XPath- The XPath expression, used to navigate to the tag.
  • processors – input processor name. If any processor, is not defined, then, default one is called.
• add_css() – This method, takes the item field, and, corresponding CSS expression for it. It mainly accepts parameters as –
  • field_name – The item field name, defined in the ‘items.py’ class.
  • CSS- The CSS expression, used to navigate to the tag.
  • processors – input processor name. If any processor, is not defined, then the default one is called.
• add_value() – This method, takes string literal, and, its value. It accepts parameters as –
  • field_name- any string literal.
  • value – The value of the string literal.
  • processors – input processor name. If any processor, is not defined, then the default one is called.

One can make use, of any of the above loader methods. In this article, we have used XPath expressions, to scrape data, hence the add_xpath() method, of the loader is used. In the Scrapy configuration, the processors.py file, is present, from which we can import, all mentioned processors.

Item Loader Objects

We get an item loader object, by instantiating, the ItemLoader class. The ItemLoader class, present in the Scrapy library, is the scrapy.loader.ItemLoader. The parameters, for ItemLoader object creation, are –

• item – This is the Item class, to populate,  by calling add_xpath(), add_css() or add_value() methods.
• selector – It is the, CSS or XPath expression selector, used to get data, from the website, to be scraped.
• response – Using default_selector_class, it is used to prepare a selector.

Following are the methods available for ItemLoader objects:

Sr. No	Method	Description
1	get_value(value,*processors,**kwargs)	The value is processed by the mentioned processor, and, keyword arguments. The keyword argument parameter can be :  ‘re’, A regular expression to use, for getting data, from the given value, applied before the processor.
2	add_value(fieldname,*processors, **kwargs)	Process, and, then add the given value, for the field given. Here, value is first passed, through the get_value(), by giving the processor and kwargs. It is then passed, through the field input processor. The result is appended, to the data collected, for that field. If field, already contains data, then, new data is added. The field name can have None value as well. Here, multiple values can be added, in the form of dictionary objects.
3	replace_value(fieldname, *processors, **kwargs)	This method, replaces the collected value with a new value, instead of adding it.
4	get_xpath( XPath,*processors, **kwargs)	This method receives an XPath expression. This expression is used to get a list of Unicode strings, from the selector, which is related, to the ItemLoader. This method, is similar to ItemLoader.get_value().  The parameters, of this method, are – XPath – the XPath expression to extract data from the webpage re – A regular expression string, or, a pattern to get data from the XPath region.
5	add_xpath(xpath,*processors, **kwargs)	This method, receives an XPath expression, that is used to select, a list of strings, from the selector, related to the ItemLoader. It is similar to ItemLoader.add_value(). Parameter is – XPath – The XPath expression to extract data from.
6	replace_xpath(fieldname, XPath,*processors,**kwargs)	Instead of, adding the extracted data, this method, replaces the collected data.
7	get_css(CSS, *processors, **kwargs)	This method receives a CSS selector, and, not a value, which is then used to get a list of Unicode strings, from the selector, associated with the ItemLoader. The parameters can be – CSS – The string selector to get data from re – A regular expression string or a pattern to get data from the CSS region.
8	add_css(fieldname, css, *processors, **kwargs)	This method, adds a CSS selector, to the field. It is similar to add_value(), but, receives a CSS selector. Parameter is – CSS – A string CSS selector to extract data from
9	replace_css(fieldname, CSS, *processors, **kwargs)	Instead of, adding collected data, this method replaces it, using the CSS selector.
10	load_item()	This method is used to populate, the item received so far, and return it. The data is first passed through, the output_processors, so that the final value, is assigned to each field.
11	nested_css(css, **context)	Using CSS selector, this method is used to create nested selectors. The CSS supplied, is applied relative, to the selector, associated with the ItemLoader.
12	nested_xpath(xpath)	Using the XPath selector, create a nested loader. The XPath supplied, is applied relative, to the selector associated with the ItemLoader.

Nested Loaders

Nested loaders are useful when we are parsing values, that are related, from the subsection of a document. Without them, we need to mention the entire XPath or CSS path, of the data we want to extract. Consider, the following HTML footer example  –

Python3

# Create loader object
loader = ItemLoader(item=Item())

# Item loader method for phoneno,
# mention the field name and xpath expression
loader.add_xpath('phoneno',
                 '//footer/a[@class = "phoneno"]/@href')

# Item loader method for map,
# mention the field name and xpath expression
loader.add_xpath('map',
                 '//footer/a[@class = "map"]/@href')

# populate the item
loader.load_item()

Using nested loaders, we can avoid, using the nested footer selector, as follows: 

Python3

# Define Item Loader object by passing item
loader = ItemLoader(item=Item())

# Create nested loader with footer selector
footer_loader = loader.nested_xpath('//footer')

# Add phoneno xpath values relative to the footer
footer_loader.add_xpath('phoneno', 'a[@class = "phoneno"]/@href')

# Add map xpath values relative to the footer
footer_loader.add_xpath('map', 'a[@class = "map"]/@href')

# Call loader.load_item() to populate values
loader.load_item()

Please note the following points about nested loaders:

• They work with CSS and XPath selectors.
• They can be nested randomly.
• They can make the code look simpler.
  • Do not use them needlessly, else the parser can get difficult to read.

Reusing and Extending Item Loaders

Maintenance, becomes difficult, as the project grows, and, also the number of spiders, written for data scraping. Also, the parsing rules may change, for every other spider. To simplify the maintenance, of parsing, Item Loaders support, regular Python inheritance, to deal with differences, present in a group of spiders. Let us look, at an example, where extending loaders, may turn beneficial.

Suppose, any eCommerce book website, has its book author names, starting with an “*”(asterisk). If you want, to remove those “*”, present in the final scraped author names, we can reuse, and, extend the default loader class ‘BookLoader’ as follows:

Python3

# Import the MapCompose built-in processor
from itemloaders.processors import MapCompose

# Import the existing BookLoader
# Item loader used for scraping book data
from myproject.ItemLoaders import BookLoader

# Custom function to remove the '*'
def strip_asterisk(x):
    return x.strip('*')

# Extend and reuse the existing BookLoader class
class SiteSpecificLoader(BookLoader):
    authorname = MapCompose(strip_asterisk,
                            BookLoader.authorname)

In the above code, the BookLoader is a parent class, for the SiteSpecificLoader class. By reusing the existing loader, we have added only the strip “*” functionality, in the new loader class.

Declaring Custom Item Loaders Processors

Just like Items, Item Loaders too can be declared by using the class syntax.  The declaration can be done, as follows:

Python3

# Import the Item Loader class
from scrapy.loader import ItemLoader

# Import the processors
from scrapy.loader.processors import TakeFirst, MapCompose, Join

# Extend the ItemLoader class
class BookLoader(ItemLoader):

    # Mention the default output processor
    default_output_processor = Takefirst()

    # Input processor for book name
    book_name_in = MapCompose(unicode.title)

    # Output processor for book name
    book_name_out = Join()

    # Input processor for book price
    book_price_in = MapCompose(unicode.strip)

The code can be understood as:

• The BookLoader class extends the ItemLoader.
• The book_name_in, has a MapCompose instance, with defined function unicode.title, that would get applied on the book_name item.
• The book_name_out is defined as Join() class instance.
• The book_price_in, has a MapCompose instance, with a defined function unicode.strip, that would get applied on the book_price item.

Implementing Item Loaders to Parse Data:

Now, we have a general understanding of Item Loaders. Let us implement, the above concepts, in our example –

• In the spider ‘gfg_loadbookdata.py’ file, we define ItemLoaders, by making use of  Scrapy.Loader.Itemloader module. The syntax will be -“from scrapy.loader import ItemLoader”.
• In the parse method, which is the default callback method of the spider, we are already looping through all the books.
• Inside the loop, create an object of ItemLoader class, by using the arguments as –
  • Pass the item attribute name, as GfgItemloadersItem
  • Pass selector attribute, as ‘books’
  • So the code will look –  “loader = ItemLoader(item=GfgItemloadersItem(), selector=books)”
• Use the Item loader method, add_xpath(), and, pass the item field name, and, XPath expression.
• Use ‘price’ field, and, write its XPath in the add_xpath() method. Syntax will be – “loader.add_xpath(‘price’, ‘.//*[@class=”product_price”]/p/text()’)”. Here, we are selecting, the text of the price, by navigating till the price tag, and, then fetching the  using the text() method.
• Use ‘title’ field, and write its XPath expression, in the add_xpath() method. Syntax will be – “loader.add_xpath(‘title’, ‘.//h3/a/@title’)”. Here, we are fetching, the value of the ‘title’ attribute, of the <a> tag.
• Yield, the loader item, now by using the load_item(), method of the loader.
• Now, let us make changes, in the ‘items.py’ file. For Every Item field, defined here, there is an input and output processor. When data is received, the input processor acts upon them, as defined by the function. Then, a list of internal elements is prepared, and passed to the output processor function, when they are populated, using the load_item() method. Currently, price and title are defined, as scrapy.Field().
• For the Book Price values, we need to replace the ‘£’ sign with a blank. Here,  we assign, MapCompose() built-in processor, as an input_processor. The first parameter to this is the remove_tags method, which removes all the tags, present in the selected response. The second parameter will be our custom function, remove_pound_sign(), that will replace ‘£’ sign a blank. The output_processor, for the Price field, will be TakeFirst(), which is the built-in processor, used to return the first non-null value, from the output. Hence, the syntax for the Price Item field will be price = scrapy.Field(input_processor=MapCompose(remove_tags, remove_pound_sign), output_processor=TakeFirst()).
• The functions, used for Price, are remove_tags and remove_pound_sign. The remove_tags() method, is imported from the Urllib HTML module. It removes, all the tags present, in the scraped response. The remove_pound_sign(), is our custom method that accepts the ‘price’ value of every book, and, replaces it with a blank. The inbuilt Python, replace function, is used for the replacement.
• Similarly, for the Book Title, we will replace ‘&’ with ‘AND’, by assigning appropriate Input and Output processors. The input_processor will be MapCompose(), the first parameter to which, will be the remove_tags method, which will remove all the tags, and, replace_and_sign(), our custom method to replace ‘&’ with ‘AND’. The output_processor will be TakeFirst() that will return, the first non-null value, from the output. Hence, the book title field will be title= scrapy.Field(input_processor=MapCompose(remove_tags, replace_and_sign), output_processor=TakeFirst()).
• The functions, used for Title, are remove_tags and replace_and_sign. The remove_tags method is imported from the Urllib HTML module. It removes all the tags, present, in the scraped response. The replace_and_sign(), is our custom method, that accepts the ‘&’ operator, of every book, and, replaces it with a ‘AND’. The inbuilt Python, replace function, is used for the replacement.

The final code, for our ‘items.py’ class, will look as shown below:

Python3

# Define here the models for your scraped items

# import Scrapy library
import scrapy

# import itemloader methods
from itemloaders.processors import TakeFirst, MapCompose

# import remove_tags method to remove all tags present
# in the response
from w3lib.html import remove_tags

# custom method to replace '&' with 'AND'
# in book title
def replace_and_sign(value):

    # python replace method to replace '&' operator
    # with 'AND'
    return value.replace('&', ' AND ')

# custom method to remove the pound currency sign from
# book price
def remove_pound_sign(value):

    # for pound press Alt + 0163
    # python replace method to replace '£' with a blank
    return value.replace('£', '').strip()

# Item class to define all the Item fields - book title
# and price
class GfgItemloadersItem(scrapy.Item):

    # Assign the input and output processor for book price field
    price = scrapy.Field(input_processor=MapCompose(
        remove_tags, remove_pound_sign), output_processor=TakeFirst())

    # Assign the input and output processor for book title field
    title = scrapy.Field(input_processor=MapCompose(
        remove_tags, replace_and_sign), output_processor=TakeFirst())

The final spider file code will look as follows:

Python3

# Import the required Scrapy library
import scrapy

# Import the Item Loader library
from scrapy.loader import ItemLoader

# Import the items class from 'items.py' file
from ..items import GfgItemloadersItem

# Spider class having Item loader
class GfgLoadbookdataSpider(scrapy.Spider):
    # Name of the spider
    name = 'gfg_loadbookdata'

    # The domain  to be scraped
    allowed_domains = [
        'books.toscrape.com/catalogue/category/books/womens-fiction_9']

    # The webpage to be scraped
    start_urls = [
        'http://books.toscrape.com/catalogue/category/books/womens-fiction_9/']

    # Default callback method used by the spider
    # Data in the response will be processed here
    def parse(self, response):

      # Loop through all the books using XPath expression
        for books in response.xpath('//*[@class="product_pod"]'):

            # Define Item Loader object,
            # by passing item and selector attribute
            loader = ItemLoader(item=GfgItemloadersItem(), selector=books)

            # Item loader method add_xpath(),for price,
            # mention the field name and xpath expression
            loader.add_xpath('price', './/*[@class="product_price"]/p/text()')

            # Item loader method add_xpath(),
            # for title, mention the field name
            # and xpath expression
            loader.add_xpath('title', './/h3/a/@title')

            # use the load_item method of
            # loader to populate the parsed items
            yield loader.load_item()

We can run, and, save the data in JSON file, using the scrapy ‘crawl’ command using the syntax scrapy crawl spider_name as –

scrapy crawl gfg_loadbookdata -o parsed_bookdata.json

The above command will scrape the data, parse the data, which means the pound sign, won’t be there, and, ‘&’ operator will be replaced with ‘AND’. The  parsed_bookdata.json file is created as follows:

The parsed JSON output  file using Item Loaders