Skip to content
Related Articles

Related Articles

Improve Article
Save Article
Like Article

How to get the Daily News using Python

  • Last Updated : 14 Sep, 2021

In this article, we are going to see how to get daily news using Python. Here we will use Beautiful Soup and the request module to scrape the data.

Modules needed

  • bs4: Beautiful Soup(bs4) is a Python library for pulling data out of HTML and XML files. This module does not come built-in with Python. To install this type the below command in the terminal.
pip install bs4
  • requests: Request allows you to send HTTP/1.1 requests extremely easily. This module also does not come built-in with Python. To install this type the below command in the terminal.
pip install requests

Stepwise Implementation:

Step 1: First of all, make sure to import these libraries.

 Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.  

To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning - Basic Level Course

Python3






import requests
from bs4 import BeautifulSoup

Step 2: Then to get the HTML contents of https://www.bbc.com/news, add these 2 lines of code:

Python3




response = requests.get(url)

Step 3: Get specific HTML tag

In order to find the HTML tags within which news headlines are contained, head over to https://www.bbc.com/news and inspect a news headline by right-clicking it and clicking “inspect”:

You will see that all headlines are contained within “<h3>” tags. Therefore, to scrape all “<h3>” tags within this webpage, add these lines of code to your script:

First, we define “soup” as the HTML content of the BBC news webpage. Next, we define “headlines” as an array of all “<h3>” tags found within the webpage. Finally, the script paddles through the “headlines” array and displays all of its contents one by one, ridding each element of its outerHTML and displaying only its text contents using the “text.strip()” method.

Python3






soup = BeautifulSoup(response.text, 'html.parser')
headlines = soup.find('body').find_all('h3')
for x in headlines:
    print(x.text.strip())

Below is the implementation:

Python3




import requests
from bs4 import BeautifulSoup
  
response = requests.get(url)
  
soup = BeautifulSoup(response.text, 'html.parser')
headlines = soup.find('body').find_all('h3')
for x in headlines:
    print(x.text.strip())

Output:

Cleaning the data

You might have noticed that your output contains duplicate news headlines and text contents that aren’t news headlines.

Create a list of all the text elements you want to get rid of:

unwanted = [‘BBC World News TV’, ‘BBC World Service Radio’, ‘News daily newsletter’, ‘Mobile app’, ‘Get in touch’]

Then print text elements only if they are not in this list by putting:

print(x.text.strip())

Below is the implementation:

Python3




import requests
from bs4 import BeautifulSoup
  
response = requests.get(url)
  
soup = BeautifulSoup(response.text, 'html.parser')
headlines = soup.find('body').find_all('h3')
unwanted = ['BBC World News TV', 'BBC World Service Radio',
            'News daily newsletter', 'Mobile app', 'Get in touch']
  
for x in list(dict.fromkeys(headlines)):
    if x.text.strip() not in unwanted:
        print(x.text.strip())

Output:




My Personal Notes arrow_drop_up
Recommended Articles
Page :