N-grams are contiguous sequences of n items (words, characters, or symbols) extracted from a given sample of text or speech. They are widely used in natural language processing (NLP) and computational linguistics for various applications such as language modelling, text generation, and information retrieval. In R Programming Language the concept of n-grams is a fundamental building block in understanding the structure and patterns within textual data.
Types of N-grams:
The types of N-grams are determined by the value of ‘n’. Here are some common types:
-
Unigrams (1-grams): Single words, representing the most basic units of text.
Probability of a unigram P(w_i):
-
Bigrams (2 grams): Pairs of consecutive words. Example: “natural language,” “processing model.”
Probability of a bigram P(w_i | w_{i-1}):
-
Trigrams (3-grams): Triplets of consecutive words. Example: “contiguous sequences of,” “pairs of consecutive.”
Probability of a trigram P(w_i | w_{i-2}, w_{i-1}):
Consider the sentence: “I love natural language processing.”
- Unigrams: “I,” “love,” “natural,” “language,” “processing.”
- Bigrams: “I love,” “love natural,” “natural language,” “language processing.”
- Trigrams: “I love natural,” “love natural language,” “natural language processing.”
Tokenization:
- Before generating n-grams, the text is typically tokenized, breaking it into individual words or characters.
- Tokenization is a crucial step that defines the units from which n-grams are extracted.
Applications of N-grams
- Language Modeling: N-grams are used to model the likelihood of a sequence of words in a language. This is foundational to machine translation, speech recognition, and other language generation tasks.
- Text Prediction: Given a sequence of n-1 words, n-grams can be used to predict the next word in a sentence.
- Information Retrieval: In search engines, n-grams are used to index and retrieve documents based on user queries.
- Text Classification: N-grams are used as features in text classification tasks, helping to capture local word patterns.
N-gram Probability
- The probability of an n-gram occurring is often estimated from the frequency of its occurrence in a given corpus.
- For example, the probability of a bigram “word1 word2” is estimated as the count of occurrences of the sequence divided by the total number of bigrams in the corpus.
N-gram Smoothing
- In practice, some n-grams may not be present in the training corpus. Smoothing techniques, such as add-one (Laplace) smoothing, are applied to handle unseen n-grams.
- Smoothing ensures that all n-grams have non-zero probabilities.
Limitations
- N-grams have limitations, especially with longer sequences, as they may not capture long-range dependencies in language effectively.
- Techniques like neural language models, such as LSTMs and Transformers, are employed to address these limitations.
Implimentation of N-Grams using R
Create dataset and tokenization
R
# Install and load the ngram packageinstall.packages("ngram")
library(ngram)
# Example text corpuscorpus <- c(
"This is the first document.",
"The second document is here.",
"And this is the third one.",
"This, is an example sentence!")# Convert to lowercasecorpus_lower <- tolower(corpus)
cat(corpus_lower)
|
Output:
this is the first document. the second document is here. and this is the third one. this, is an example sentence!
Beg of N-Grams
Here we are converting text into unigrams,bigrams and trigrams.
R
# Tokenize into unigrams for the entire textunigrams_text <- ngram(corpus_lower, n = 1,sep = ".!,/ ")
print(unigrams_text)
# Extract Unigram valuesunigram_values <- get.ngrams(unigrams_text)
print(unigram_values)
# Tokenize into bigrams for the entire textbigrams_text <- ngram(corpus_lower, n = 2, sep = ".!,/ ")
print(bigrams_text)
bigram_values <- get.ngrams(bigrams_text)
print(bigram_values)
# Tokenize into trigrams for the entire texttrigrams_text <- ngram(corpus_lower, n = 3, sep = ".!,/ ")
print(trigrams_text)
trigram_values <- get.ngrams(trigrams_text)
print(trigram_values)
|
Output:
An ngram object with 13 1-grams
[1] "the" "an" "example" "and" "first" "document"
[7] "is" "second" "one" "sentence" "this" "third"
[13] "here"
An ngram object with 14 2-grams
[1] "document is" "and this" "is the" "third one"
[5] "this is" "first document" "the third" "example sentence"
[9] "an example" "second document" "is an" "the second"
[13] "the first" "is here"
An ngram object with 12 3-grams
[1] "the first document" "this is the" "the second document"
[4] "second document is" "and this is" "document is here"
[7] "this, is a" "is the third" "an example sentence"
[10] "the third one" "is the first" "is an example"
In this example, the sep = ” ” argument is used to specify space as the separator for creating bigrams. we can adjust the sep parameter to include any other characters that we want to use as separators between words.
Probabilities of each N-Grams
R
# Calculate probabilities for unigramsunigram_probabilities <- get.phrasetable(unigrams_text)
cat("Unigram Probabilities:\n")
print(unigram_probabilities)
# Calculate probabilities for bigramsbigram_probabilities <- get.phrasetable(bigrams_text)
cat("\nBigram Probabilities:\n")
print(bigram_probabilities)
# Calculate probabilities for trigramstrigram_probabilities <- get.phrasetable(trigrams_text)
cat("\nTrigram Probabilities:\n")
print(trigram_probabilities)
|
Output:
Unigram Probabilities:
ngrams freq prop
1 is 4 0.19047619
2 the 3 0.14285714
3 this 3 0.14285714
4 document 2 0.09523810
5 an 1 0.04761905
6 example 1 0.04761905
7 and 1 0.04761905
8 first 1 0.04761905
9 second 1 0.04761905
10 one 1 0.04761905
11 sentence 1 0.04761905
12 third 1 0.04761905
13 here 1 0.04761905
Bigram Probabilities:
ngrams freq prop
1 this is 3 0.17647059
2 is the 2 0.11764706
3 document is 1 0.05882353
4 and this 1 0.05882353
5 third one 1 0.05882353
6 first document 1 0.05882353
7 the third 1 0.05882353
8 example sentence 1 0.05882353
9 an example 1 0.05882353
10 second document 1 0.05882353
11 is an 1 0.05882353
12 the second 1 0.05882353
13 the first 1 0.05882353
14 is here 1 0.05882353
Trigram Probabilities:
ngrams freq prop
1 this is the 2 0.15384615
2 the first document 1 0.07692308
3 the second document 1 0.07692308
4 second document is 1 0.07692308
5 and this is 1 0.07692308
6 document is here 1 0.07692308
7 this is an 1 0.07692308
8 is the third 1 0.07692308
9 an example sentence 1 0.07692308
10 the third one 1 0.07692308
11 is the first 1 0.07692308
12 is an example 1 0.07692308
Conclusion
N-grams are a versatile and widely used concept in NLP, providing a flexible way to represent and analyze the structure of textual data. They form the basis for more advanced language models and applications in the field of natural language processing.