Open In App

Extract JSON from HTML using BeautifulSoup in Python

Improve
Improve
Like Article
Like
Save
Share
Report

In this article, we are going to extract JSON from HTML using BeautifulSoup in Python.

Module needed

  • bs4: Beautiful Soup(bs4) is a Python library for pulling data out of HTML and XML files. This module does not come built-in with Python. To install this type the below command in the terminal.
pip install bs4
  • requests: Request allows you to send HTTP/1.1 requests extremely easily. This module also does not come built-in with Python. To install this type the below command in the terminal.
pip install requests

Approach:

  • Import all the required modules.
  • Pass the URL in the get function(UDF) so that it will pass a GET request to a URL, and it will return a response.

Syntax: requests.get(url, args)

  • Now Parse the HTML content using bs4.

Syntax: BeautifulSoup(page.text, ‘html.parser’)

Parameters:

  • page.text : It is the raw HTML content.
  • html.parser : Specifying the HTML parser we want to use.
  • Now get all the required data with find() function.

Now find the customer list with li, a, p tag where some unique class or id. You can open the webpage in the browser and inspect the relevant element by pressing right-click as shown in the figure.

  • Create a Json file and use json.dump() method to convert python objects into appropriate JSON objects.

Below is the full implementation:

Python3




# Import the required modules
import requests
from bs4 import BeautifulSoup
import json
 
 
# Function will return a list of dictionaries
# each containing information of books.
def json_from_html_using_bs4(base_url):
 
    # requests.get(url) returns a response that is saved
    # in a response object called page.
    page = requests.get(base_url)
 
    # page.text gives us access to the web data in text
    # format, we pass it as an argument to BeautifulSoup
    # along with the html.parser which will create a
    # parsed tree in soup.
    soup = BeautifulSoup(page.text, "html.parser")
 
    # soup.find_all finds the div's, all having the same
    # class "col-xs-6 col-sm-4 col-md-3 col-lg-3" that is
    # stored in books
    books = soup.find_all(
        'li', attrs={'class':
                'col-xs-6 col-sm-4 col-md-3 col-lg-3'})
 
    # Initialise the required variables
    star = ['One', 'Two', 'Three', 'Four', 'Five']
    res, book_no = [], 1
     
    # Iterate books classand check for the given tags
    # to get the information of each books.
    for book in books:
 
        # Title of book in <img> tag with "alt" key.
        title = book.find('img')['alt']
 
        # Link of book in <a> tag with "href" key
        link = base_url[:37] + book.find('a')['href']
 
        # Rating of book from
 
<p> tag
        for index in range(5):
            find_stars = book.find(
            'p', attrs={'class': 'star-rating ' + star[index]})
             
            # Check which star-rating class is not
            # returning None and then break the loop
            if find_stars is not None:
                stars = star[index] + " out of 5"
                break
 
        # Price of book from
 
<p> tag in price_color class
        price = book.find('p', attrs={'class': 'price_color'
                                                    }).text
 
        # Stock Status of book from
 
<p> tag in
        # instock availability class.
        instock = book.find('p', attrs={'class':
                        'instock availability'}).text.strip()
         
        # Create a dictionary with the above book information
        data = {'book no': str(book_no), 'title': title,
            'rating': stars, 'price': price, 'link': link,
            'stock': instock}
 
        # Append the dictionary to the list
        res.append(data)
        book_no += 1
    return res
 
 
# Main Function
if __name__ == "__main__":
 
    # Enter the url of website
 
    # Function will return a list of dictionaries
    res = json_from_html_using_bs4(base_url)
 
    # Convert the python objects into json object and export
    # it to books.json file.
    with open('books.json', 'w', encoding='latin-1') as f:
        json.dump(res, f, indent=8, ensure_ascii=False)
    print("Created Json File")


Output:

Created Json File

Our JSON file output:



Last Updated : 16 Dec, 2021
Like Article
Save Article
Previous
Next
Share your thoughts in the comments
Similar Reads