Python provides a library called BeautifulSoap to easily allow web scraping. BeautifulSoup object is provided by Beautiful Soup which is a web scraping framework for Python. Web scraping is the process of extracting data from the website using automated tools to make the process faster. The BeautifulSoup object represents the parsed document as a whole. In this article, we’ll be scrapping a simple website and replacing the content in the parsed “soup” variable.
For the purpose of this article, let’s create a virtual environment (venv) as it helps us to manage separate package installations for different projects and to avoid messing up with dependencies and interpreters!
More about, how to create a virtual environment can be read from here: Create a virtual environment
Creating a virtual environment
Navigate to your project directory and run this command to create a virtual environment named “env” in your project directory.
python3 -m venv env
Activate the “env” by typing.
Having interpreter activated, we can see the name of an interpreter in our command line before :~$ symbol
Installing required modules
- BeautifulSoup: A library to scrape the web pages.
pip install bs4
- requests: This makes the process of sending HTTP requests.
pip install requests
- Let’s start by importing libraries and storing “GET” requests response in a variable.
A status of 200 implies a successful request.
- Now let’s parse the content as a BeautifulSoup object to extract the title and header tags of the website (as for this article) and to replace it in the original soup variable. The find() method returns the first matching case from the soup object.
Replacing the content of the parsed soup obj with the “.string” method.
Thus, the title tag and heading tags have been replaced in the original soup variable.
Note: We can’t push the modified page back to the website as those pages are rendered from servers where they are hosted.
Below is the complete program:
Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.
To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course.