Project Idea – Searching news from Old Newspaper using NLP
We know that the newspaper is an enriched source of knowledge. When a person needs some information about a particular topic or subject he searches online, but it is difficult to get all old news articles from regional local newspapers related to our search. As not every local newspaper provides an online search for people.In this article, we will present an idea to overcome this problem.
What project does?
- This project uses images or pdf of newspaper images from old regional newspapers as input for the database.
- The model will extract the text from images using Pytesseract.
- The text from the Pytesseract would be cleaned by NLP practices to simplify and eliminate the words (stop words) that are not helpful for us.
- The data will be saved in the form of key-value pair in which keys have an image path and values have keywords in the image.
- Searching: When the user visits the website he will type the topic name or entity name in the search box then images of the newspaper will load on the screen.
Why NLP ?
Newspaper articles contain many articles, prepositions, and other stop words that are not useful to us, so NLP helps us to remove those stop words. It also helps to get unique words.
Technologies used :
Tools used :
- Google colab
- pytesseract: image to text.
- NLTK: text pre-processing, filtering.
- pandas: storing dataframe.
Use Case Diagram
Step By step Implementation:
First, Install required libraries on colab.
Let’s import all the necessary libraries:
This will clean the text to get important names, keywords, etc. Stop words and duplicate words are removed by the below function.
when given image path as a parameter it returns preprocessed text in the text variable. then this text is passed as a parameter to pre(). this function returns a dictionary with filename and important text.
here we are defining the dataframe to store the dictionary which has an image path and the text inside the image. We will use this dataframe for searching.
Listing all images in the content folder.
For loop to get all news images from the folder.
Processing the images
Saving the dataframe to database.
Saving the dataframe
Open the dataframe file from storage.
We provide a string as input for the function to get an image in which the keyword is present.
Showing the result
We have searched for the word ‘modi‘. The first newspaper which has our searched word in it so it’s shown here.
Scope for Improvement
We could use a dedicated database, like lucent or elastic search to make the search more efficient and fast. But for the time being, we use the pandas library to get the path of the image to display to the user.
Project Application in Real-Life
- Voter Helper: As elections are coming and Aman is a voter who doesn’t know that much about the politician whom he’s going to vote for. In this situation, he opens our local news.search.in, then he searches for the politician. The website will show the no of the article from the national newspapers as well as from the regional local newspapers, related to the searched person. Now he is ready to decide his vote.
- Student Research made easy: It can be useful for students who are researching for the topic, as this will give every article from all newspapers related to their topic in image form so they can make notes out of it.
- Search Engine for Newspaper company: It can be used for press companies to have a search feature on their website.