Flipkart Reviews Sentiment Analysis using Python

This article is based on the analysis of the reviews and ratings user gives on Flipkart to make others aware of their experience and moreover about the quality of the product and brand. So, by analyzing that data we can tell the users a lot about the products and also the ways to enhance the quality of the product. 

Today we will be using Machine Learning to analyze that data and make it more efficient to understand and prediction ready.

Our task is to predict whether the review given is positive or negative.

Before starting the code, download the dataset by clicking this link.

Importing Libraries and Datasets

The libraries used are : 

• Pandas : For importing the dataset.
• Scikit-learn : For importing the model, accuracy module, and TfidfVectorizer.
• Warning : To ignore all the warnings
• Matplotlib : To plot the visualization. Also used Wordcloud for that.
• Seaborn : For data visualization.

Python3

import warnings
warnings.filterwarnings('ignore')
import pandas as pd
import re
import seaborn as sns
from sklearn.feature_extraction.text import TfidfVectorizer
import matplotlib.pyplot as plt
from wordcloud import WordCloud

For text analysis, we will be using NLTK Library. From that we will be requiring stopword. So let’s download and import it using the below command.

Python3

import nltk
nltk.download('stopwords')
from nltk.corpus import stopwords

After that import the downloaded dataset using the below code.

Python3

data = pd.read_csv('flipkart_data.csv')
data.head()

Output :

Preprocessing and cleaning the reviews 

As, the real data is multi-labelled, so firstly explore those labels then we will convert them into 2 classes.

Python3

# unique ratings
pd.unique(data['rating'])

Output:

array([5, 4, 1, 3, 2], dtype=int64)

Let’s see the countplot for the same.

Python3

sns.countplot(data=data,
              x='rating',
              order=data.rating.value_counts().index)

Output :

To predict the Sentiment as positive(numerical value = 1) or negative(numerical value = 0), we need to change the rating column into an another column of 0s and 1s category. For that the condition will be like if the rating is less than or equal to 4, then it is negative(0) else positive(1). For better understanding, refer the code below.

Python3

# rating label(final)
pos_neg = []
for i in range(len(data['rating'])):
    if data['rating'][i] >= 5:
        pos_neg.append(1)
    else:
        pos_neg.append(0)

data['label'] = pos_neg

Let’s create the function to preprocess the dataset 

Python3

from tqdm import tqdm


def preprocess_text(text_data):
    preprocessed_text = []

    for sentence in tqdm(text_data):
        # Removing punctuations
        sentence = re.sub(r'[^\w\s]', '', sentence)

        # Converting lowercase and removing stopwords
        preprocessed_text.append(' '.join(token.lower()
                                          for token in nltk.word_tokenize(sentence)
                                          if token.lower() not in stopwords.words('english')))

    return preprocessed_text

Now, we can implement this function for the dataset. The code for that is given below.

Python3

preprocessed_review = preprocess_text(data['review'].values)
data['review'] = preprocessed_review

Once we have done with the preprocess. Let’s see the top 5 rows to see the improved dataset.

Python3

data.head()

Output :

Analysis of the Dataset

Let’s check out that how many counts are there for positive and negative sentiments.

Python3

data["label"].value_counts()

Output : 

1    5726
0    4250

To have the better picture of the importance of the words let’s create the Wordcloud of all the words with label = 1 i.e. positive

Python3

consolidated = ' '.join(
    word for word in data['review'][data['label'] == 1].astype(str))
wordCloud = WordCloud(width=1600, height=800,
                      random_state=21, max_font_size=110)
plt.figure(figsize=(15, 10))
plt.imshow(wordCloud.generate(consolidated), interpolation='bilinear')
plt.axis('off')
plt.show()

Output :

Now it’s clear that the words like good, nice, product have high frequency in positive review, which satisfies our assumptions.

Let’s create the vectors.

Converting text into Vectors

TF-IDF calculates that how relevant a word in a series or corpus is to a text. The meaning increases proportionally to the number of times in the text a word appears but is compensated by the word frequency in the corpus (data-set). We will be implementing this with the code below.

Python3

cv = TfidfVectorizer(max_features=2500)
X = cv.fit_transform(data['review'] ).toarray()

To Print the X generated

Python3

X

Output:

array([[0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       ...,
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.]])

Model training, Evaluation, and Prediction

Once analysis and vectorization is done. We can now explore any machine learning model to train the data. But before that perform the train-test split.

Python3

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, data['label'],
                                                    test_size=0.33,
                                                    stratify=data['label'],
                                                    random_state = 42)

Now we can train any model, Let’s explore the Decision Tree for the prediction.

Python3

from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(random_state=0)
model.fit(X_train,y_train)

#testing the model
pred = model.predict(X_train)
print(accuracy_score(y_train,pred))

Output :

0.9244351339218914

Let’s see the confusion matrix for the results.

Python3

from sklearn import metrics
cm = confusion_matrix(y_train,pred)

cm_display = metrics.ConfusionMatrixDisplay(confusion_matrix = cm,
                                            display_labels = [False, True])

cm_display.plot()
plt.show()

Output : 

Conclusion

Decision Tree Classifier is performing well with this data. In future, we can also work with large data by scraping it through the website.