Bag-Of-Words Model In R

Effectively representing textual data is crucial for training models in Machine Learning. The Bag-of-Words (BOW) model serves this purpose by transforming text into numerical form. This article comprehensively explores the Bag-of-Words model, elucidating its fundamental concepts and utility in text representation for Machine Learning.

What is Bag-of-Words?

Bag-of-words is useful for representing textual data in a passage when using text for training and modelling in Machine Learning. We represent the text in the form of numbers generally in Machine Learning. BOW allows to extract features from text using numerous ways to convert text into numbers. It provides two main features:

1. Known vocabulary of words: BOW relies on a predefined vocabulary of words. Each unique word in the corpus is assigned a unique identifier.
2. Frequency or probability of occurrence of words: BOW captures the frequency or probability of occurrence of each word in a document. This information is used to create a numerical representation of the text.

With the help of a bag of words, we can detect the type of document, useful for sentimental analysis, document classification, and spam filtering.

The BOW model treats each sentence as a vector, where each element of the vector corresponds to the frequency of a word in the dictionary converting a collection of text documents into a matrix, where each row represents a document, and each column represents a unique word.

But, BOW does not preserve the structure of sentences or consider word order. It treats each word as independent, ignoring semantic relationships.

Bag-of-Words Example

Suppose we have the following two sentences:

1. This is my car.
2. My car is red in colour.

So we would have a dictionary of some words and we track the frequency of words of each sentence.

With the frequency table, we can feed this vector into machine learning models and train them.

Text Classification using Bag of Words

We will be using the CSV file of Poems from poetryfoundation.org from kaggle.com.

Step 1: Install the libraries

install.packages("data.table")
install.packages("stringr")
install.packages("tm")
install.packages("caret")

Step 2: Import the data

R

library(data.table)
library(stringr)
library(tm)
library(slam)

data = read.csv("/kaggle/input/modern-renaissance-poetry/all.csv")$content

Preprocessing of Data

Before moving ahead, a text needs to be preprocessed before moving ahead. Here are two texts, for eg:

1. Hello ,,, how are <b>you</b>
2. I am fine, what about you?

First sentence contains lot of unnecessary characters which can make the model inaccurate. However the second sentence is quite perfect, still the comma, question mark is not required. Punctuation generally don’t add much information, similarly the case.

Second is stopwords. These words such as “and”, “in”, “on”, “the” don’t add much information and can skew the model. Hence we need to remove such words.

Step 3: Preprocess the data. It includes:

• Convert to lower case
• Remove punctuation
• remove numbers
• remove stopwords
• strip whitespace
• Finally create a matrix of words to document

We will be using tm and slam used for text mining and matrix usage.

R

corpus = Corpus(VectorSource(data))

corpus = tm_map(corpus, content_transformer(tolower))
corpus = tm_map(corpus, removePunctuation)
corpus = tm_map(corpus, removeNumbers)

# removing stop words such as the, a, etc.
corpus = tm_map(corpus, removeWords, stopwords("SMART"))
# removing white space
corpus = tm_map(corpus, stripWhitespace)

Document-Term Matrix

It is the frequency table with each document on one axis, and dictionary on the other. We will create a matrix using the DocumentTermMatrix method by passing the data corpus and then convert the object into matrix.

R

matrix = as.matrix(DocumentTermMatrix(corpus))
print(matrix)

Output:

Step 4: First sum the columns of each word and then check the top ten words

R

word_frequencies = colSums(matrix)

N = 10
top_words = names(sort(word_frequencies, decreasing = TRUE)[1:N])
print(word_frequencies[top_words])

Output:

Step 5: We can also plot the word frequencies using barplot

R

top_frequencies = word_frequencies[top_words]
barplot(top_frequencies, main = "Top 10 Word Frequencies", xlab = "Words", ylab = "Frequency", col = "darkgreen")

Output:

Bag-Of-Words Model In R

In the following example, we use spam email dataset for the classification using bag of words. We use SVM classifier for classification of spam and ham(original). Download the dataset link.

Step 1: Load all required libraries

R

library(data.table) # Dataframe library
library(stringr) # For string methods
library(caret) # For confusion matrix
library(tm) # For Text Mining library
library(slam) # For preprocessing
library(e1071)  # For SVM classifier

Step 2: Load the dataset and preprocess the dataset similar to that of previous example

R

data = read.csv("/kaggle/input/spam-email/spam.csv")

corpus = Corpus(VectorSource(data$Message))
labels = as.numeric(factor(data$Category, levels = c("ham", "spam")))
labels = labels - 1
# Preprocess the text
corpus = tm_map(corpus, content_transformer(tolower))
corpus = tm_map(corpus, removePunctuation)
corpus = tm_map(corpus, removeNumbers)
corpus = tm_map(corpus, removeWords, stopwords("SMART"))
corpus = tm_map(corpus, stripWhitespace)

dtm <- DocumentTermMatrix(corpus)
matrix <- as.matrix(dtm)

#converting matrix to dataframe
dtm_df <- as.data.table(matrix)

# applying the labels that are used for ham and spam using numerical encoding
dtm_df$label <- labels
head(dtm_df)

Output:

Step 3: Perform the train test split in the ration of 80% 20% for train and test set respectively.

R

# setting randomization
set.seed(42)
#splitting the dataset into train and test in ration 80-20
split_index <- sample(1:nrow(dtm_df), 0.8 * nrow(dtm_df))
train_set <- dtm_df[split_index]
test_set <- dtm_df[-split_index]

Step 4: Train the model and create predictions. Then create confusion matrix

R

# training the model using training dataset
model <- svm(label ~ ., data = train_set, kernel = "linear")

# Make predictions on the test set
predictions <- predict(model, newdata = test_set[, -"label"])
# Evaluate the model
threshold <- 0.5

# converting svm prediction from probabilities into binary
binary_predictions <- ifelse(predictions > threshold, 1, 0)
confusion_matrix <- table(binary_predictions, test_set$label)
# printing the matrix
print(confusion_matrix)

# finding the accuracy of the model
accuracy <- sum(diag(confusion_matrix)) / sum(confusion_matrix)
print(paste("Accuracy:", accuracy))

Output:

binary_predictions   0   1
                 0 958  23
                 1   1 133
[1] "Accuracy: 0.97847533632287"

Hence, The model has a high accuracy of approximately 97.85%, indicating that it correctly predicts the class for a large proportion of instances.

Limitations to Bag-of-Words

• It loses the sequence information from the dataset. It just relies on the frequency of words appearing.
• It creates very sparse dataset since many words tend not to appear in a document.
• It ignores the context.
• It doesn’t relate the terms and hence loses the relationship among words.
• Tend to overfit since so many columns are formed with increasing vocabulary.

Conclusion

In conclusion, Bag-of-Words stands as a versatile tool for converting textual data into a format suitable for machine learning applications. While it excels in certain scenarios, its limitations, such as the loss of sequence information and sparse dataset creation, should be considered.