This article is based on the analysis of the reviews and ratings user gives on Flipkart to make others aware of their experience and moreover about the quality of the product and brand. So, by analyzing that data we can tell the users a lot about the products and also the ways to enhance the quality of the product.
Today we will be using Machine Learning to analyze that data and make it more efficient to understand and prediction ready.
Our task is to predict whether the review given is positive or negative.
Before starting the code, download the dataset by clicking this link.
Importing Libraries and Datasets
The libraries used are :
- Pandas : For importing the dataset.
- Scikit-learn : For importing the model, accuracy module, and TfidfVectorizer.
- Warning : To ignore all the warnings
- Matplotlib : To plot the visualization. Also used Wordcloud for that.
- Seaborn : For data visualization.
import warnings
warnings.filterwarnings('ignore')
import pandas as pd
import re
import seaborn as sns
from sklearn.feature_extraction.text import TfidfVectorizer
import matplotlib.pyplot as plt
from wordcloud import WordCloud
|
For text analysis, we will be using NLTK Library. From that we will be requiring stopword. So let’s download and import it using the below command.
import nltk
nltk.download('stopwords')
from nltk.corpus import stopwords
|
After that import the downloaded dataset using the below code.
data = pd.read_csv('flipkart_data.csv')
data.head() |
Output :
Preprocessing and cleaning the reviews
As, the real data is multi-labelled, so firstly explore those labels then we will convert them into 2 classes.
# unique ratings pd.unique(data['rating'])
|
Output:
array([5, 4, 1, 3, 2], dtype=int64)
Let’s see the countplot for the same.
sns.countplot(data=data,
x='rating',
order=data.rating.value_counts().index)
|
Output :
To predict the Sentiment as positive(numerical value = 1) or negative(numerical value = 0), we need to change the rating column into an another column of 0s and 1s category. For that the condition will be like if the rating is less than or equal to 4, then it is negative(0) else positive(1). For better understanding, refer the code below.
# rating label(final) pos_neg = []
for i in range(len(data['rating'])):
if data['rating'][i] >= 5:
pos_neg.append(1)
else:
pos_neg.append(0)
data['label'] = pos_neg
|
Let’s create the function to preprocess the dataset
from tqdm import tqdm
def preprocess_text(text_data):
preprocessed_text = []
for sentence in tqdm(text_data):
# Removing punctuations
sentence = re.sub(r'[^\w\s]', '', sentence)
# Converting lowercase and removing stopwords
preprocessed_text.append(' '.join(token.lower()
for token in nltk.word_tokenize(sentence)
if token.lower() not in stopwords.words('english')))
return preprocessed_text
|
Now, we can implement this function for the dataset. The code for that is given below.
preprocessed_review = preprocess_text(data['review'].values)
data['review'] = preprocessed_review
|
Once we have done with the preprocess. Let’s see the top 5 rows to see the improved dataset.
data.head() |
Output :
Analysis of the Dataset
Let’s check out that how many counts are there for positive and negative sentiments.
data["label"].value_counts()
|
Output :
1 5726 0 4250
To have the better picture of the importance of the words let’s create the Wordcloud of all the words with label = 1 i.e. positive
consolidated = ' '.join(
word for word in data['review'][data['label'] == 1].astype(str))
wordCloud = WordCloud(width=1600, height=800,
random_state=21, max_font_size=110)
plt.figure(figsize=(15, 10))
plt.imshow(wordCloud.generate(consolidated), interpolation='bilinear')
plt.axis('off')
plt.show() |
Output :
Now it’s clear that the words like good, nice, product have high frequency in positive review, which satisfies our assumptions.
Let’s create the vectors.
Converting text into Vectors
TF-IDF calculates that how relevant a word in a series or corpus is to a text. The meaning increases proportionally to the number of times in the text a word appears but is compensated by the word frequency in the corpus (data-set). We will be implementing this with the code below.
cv = TfidfVectorizer(max_features=2500)
X = cv.fit_transform(data['review'] ).toarray()
|
To Print the X generated
X |
Output:
array([[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]])Model training, Evaluation, and Prediction
Once analysis and vectorization is done. We can now explore any machine learning model to train the data. But before that perform the train-test split.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, data['label'],
test_size=0.33,
stratify=data['label'],
random_state = 42)
|
Now we can train any model, Let’s explore the Decision Tree for the prediction.
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(random_state=0)
model.fit(X_train,y_train) #testing the model pred = model.predict(X_train)
print(accuracy_score(y_train,pred))
|
Output :
0.9244351339218914
Let’s see the confusion matrix for the results.
from sklearn import metrics
cm = confusion_matrix(y_train,pred)
cm_display = metrics.ConfusionMatrixDisplay(confusion_matrix = cm,
display_labels = [False, True])
cm_display.plot() plt.show() |
Output :
Conclusion
Decision Tree Classifier is performing well with this data. In future, we can also work with large data by scraping it through the website.