# Latent Dirichlet Allocation

• Last Updated : 06 Jun, 2021

### Topic Modeling:

Topic modeling is a way of abstract modeling to discover the abstract ‘topics’ that occur in the collections of documents. The idea is that we will perform unsupervised classification on different documents, which find some natural groups in topics. We can answer the following question using topic modeling.

• What is the topic/main idea of the document?
• Given a document, can we find another document with a similar topic?
• How do topics field change over time?

Topic modeling can help in optimizing the search process. In this article, we will be discussing Latent Dirichlet Allocation, a topic modeling process.

Attention reader! Don’t stop learning now. Get hold of all the important Machine Learning Concepts with the Machine Learning Foundation Course at a student-friendly price and become industry ready.

## Latent Dirichlet Allocation

Latent Dirichlet allocation is one of the most popular methods for performing topic modeling. Each document consists of various words and each topic can be associated with some words. The aim behind the LDA to find topics that the document belongs to, on the basis of words contains in it. It assumes that documents with similar topics will use a similar group of words. This enables the documents to map the probability distribution over latent topics and topics are probability distribution.

### Setting up Generative Model:

• Let’s suppose we have D documents using the vocabulary of V-word types. Each document consists of an N-words token (can be removed or padded ). Now, we assume K topics, this required a K-dimensional vector that represents the topic distribution for the document.
• Each topic has a V-dimensional multinomial beta_k over words with a common symmetric prior.
• For each topic 1…k:
• Draw a multinomial over words .
• For each document 1…d:
• Draw a multinomial over topics
• For each word :
• Draw a topic  with
• Draw a word W_{N_d} \sim Mult(\varphi).

### Graphical Model of LDA:

• In the above equation, the LHS represents the probability of generating the original document from the LDA machine.
• On the right side of the equation, there are 4 probability terms, the first two terms represent Dirichlet distribution and the other two represent the multinomial distribution. The first and third terms represent the distribution of topics but the second and fourth represent the word distribution. We will discuss the Dirichlet distribution first.

Dirichlet Distribution

• Dirichlet’s distribution can be defined as a probability density for a vector-valued input having the same characteristics as our multinomial parameter . It has non-zero values such that:

• The Dirichlet distribution is parameterized by the vector α, which has the same number of elements K as the multinomial parameter θ.
• We can interpret p(θ|α) as answering the question “what is the probability density associated with multinomial distribution θ, given that our Dirichlet distribution has parameter α?”.

Dirichlet distribution

• Above is the visualization of the Dirichlet distribution, for our purpose, we can assume that corners/vertices represent the topics with words inside the triangle (the word is closer to the topic if it frequently relates with it. ) or vice-versa.
• This distribution can be extended to more than 3-dimensions. For 4-dimension we can use tetrahedron and for further dimension. We can use k-1 dimensions simplex.

### Inference:

• The inference problem in LDA to compute the posterior of the hidden variables given the document and corpus parameter \alpha and \beta. That is to compute the P(

### Example:

• Let’s consider we have two categories of topics, we have a word vector for each topic consisting of some words. Following are the words that represented different topics:
• Now, we have some document, and we scan some documents for these words:
• Now, we update the above words to topics matrix using the probabilities from document matrix below.

#### Implementation

In this implementation, we use scikit-learn and pyLDAvis. For datasets, we use yelp reviews datasets that can be found on the Yelp website.

## Python3

 # install pyldavis!pip install pyldavis# imports!pip install gensim pyLDAvis! python3 -m spacy download en_core_web_sm import pandas as pdimport numpy as np import stringimport spacyimport nltk import gensimfrom gensim import corpora import matplotlib.pyplot as plt import pyLDAvisimport pyLDAvis.gensim_models nltk.download('wordnet')from nltk.corpus import wordnet as wnnltk.download('stopwords')from nltk.corpus import stopwordsimport spacy.clispacy.cli.download("en_core_web_md")import en_core_web_md# fetch yelp review dataset and clean ityelp_review = pd.read_csv('/content/yelp.csv')yelp_review.head()# print number of document and topicsprint(len(yelp_review))print("Unique Business")print(len(yelp_review.groupby('business_id')))print("Unique User")print(len(yelp_review.groupby('user_id'))) # clean the document and remove punctuationdef clean_text(text):  delete_dict = {sp_char: '' for sp_char in string.punctuation}  delete_dict[' '] =' '  table = str.maketrans(delete_dict)  text1 = text.translate(table)  textArr= text1.split()  text2 = ' '.join([w for w in textArr if ( not w.isdigit() and                                           ( not w.isdigit() and len(w)>3))])  return text2.lower()   yelp_review['text'] = yelp_review['text'].apply(clean_text)yelp_review['Num_words_text'] = yelp_review['text'].apply(lambda x:len(str(x).split())) print('-------Reviews By Stars --------')print(yelp_review['stars'].value_counts())print(len(yelp_review))print('-------------------------')max_review_data_sentence_length  = yelp_review['Num_words_text'].max() # print short review (mask = (yelp_review['Num_words_text'] < 100) & (yelp_review['Num_words_text'] >=20)df_short_reviews = yelp_review[mask]df_sampled = df_short_reviews.groupby('stars')    .apply(lambda x: x.sample(n=100)).reset_index(drop = True) print('No of Short reviews')print(len(df_short_reviews)) # function to remove stopwordsdef remove_stopwords(text):    textArr = text.split(' ')    rem_text = " ".join([i for i in textArr if i not in stop_words])    return rem_text # remove stopwords from the textstop_words = stopwords.words('english')df_sampled['text']=df_sampled['text'].apply(remove_stopwords) # perform Lemmatizationlp = en_core_web_md.load(disable=['parser', 'ner'])def lemmatization(texts,allowed_postags=['NOUN', 'ADJ']):       output = []       for sent in texts:             doc = nlp(sent)             output.append([token.lemma_                            for token in doc if token.pos_ in allowed_postags ])      return outputtext_list=df_sampled['text'].tolist()print(text_list[2])tokenized_reviews = lemmatization(text_list)print(tokenized_reviews[2]) # convert to document term frequency:dictionary = corpora.Dictionary(tokenized_reviews)doc_term_matrix = [dictionary.doc2bow(rev) for rev in tokenized_reviews] # Creating the object for LDA model using gensim libraryLDA = gensim.models.ldamodel.LdaModel # Build LDA modellda_model = LDA(corpus=doc_term_matrix, id2word=dictionary,                num_topics=10, random_state=100,                chunksize=1000, passes=50,iterations=100)# print lda topics with respect to each word of documentlda_model.print_topics() # calculate perplexity and coherenceprint('\Perplexity: ', lda_model.log_perplexity(doc_term_matrix,                                                total_docs=10000))  # calculate coherencecoherence_model_lda = CoherenceModel(model=lda_model,                                     texts=tokenized_reviews, dictionary=dictionary ,                                     coherence='c_v')coherence_lda = coherence_model_lda.get_coherence()print('Coherence: ', coherence_lda) # Now, we use pyLDA vis to visualize itpyLDAvis.sklearn.prepare(lda_tf, dtm_tf, tf_vectorizer)
Total reviews
10000
4174
Unique User
6403
--------------
-------Reviews by stars --------
4    3526
5    3337
3    1461
2     927
1     749
Name: stars, dtype: int64
10000
-------------------------
No of Short reviews
6276
-------------------------
# review and tokenized version
decided completely write place three times tried closed website posted hours open wants drive suburbs
youd better call first place cannot trusted wasted time spent hungry minutes walking disappointed vitamin
fail said

['place', 'time', 'closed', 'website', 'hour', 'open', 'drive', 'suburb', 'first', 'place', 'time', 'hungry',
'minute', 'vitamin']
---------------------------
# LDA print topics
[(0,
'0.015*"food" + 0.013*"good" + 0.010*"gelato" + 0.008*"sandwich" + 0.008*"chocolate" + 0.005*"wife" + 0.005*"next" + 0.005*"bad" + 0.005*"night" + 0.005*"sauce"'),
(1,
'0.030*"food" + 0.021*"great" + 0.019*"place" + 0.019*"good" + 0.016*"service" + 0.011*"time" + 0.011*"nice" + 0.008*"lunch" + 0.008*"dish" + 0.007*"staff"'),
(2,
'0.023*"food" + 0.023*"good" + 0.018*"place" + 0.014*"great" + 0.009*"star" + 0.009*"service" + 0.008*"store" + 0.007*"salad" + 0.007*"well" + 0.006*"pizza"'),
(3,
'0.035*"good" + 0.025*"place" + 0.023*"food" + 0.020*"time" + 0.015*"service" + 0.012*"great" + 0.009*"friend" + 0.008*"table" + 0.008*"chicken" + 0.007*"hour"'),
(4,
'0.020*"food" + 0.019*"time" + 0.012*"good" + 0.009*"restaurant" + 0.009*"great" + 0.008*"service" + 0.007*"order" + 0.006*"small" + 0.006*"hour" + 0.006*"next"'),
(5,
'0.012*"drink" + 0.009*"star" + 0.006*"worth" + 0.006*"place" + 0.006*"friend" + 0.005*"great" + 0.005*"kid" + 0.005*"drive" + 0.005*"simple" + 0.005*"experience"'),
(6,
'0.024*"place" + 0.015*"time" + 0.012*"food" + 0.011*"price" + 0.009*"good" + 0.009*"great" + 0.009*"kid" + 0.008*"staff" + 0.008*"nice" + 0.007*"happy"'),
(7,
'0.028*"place" + 0.019*"service" + 0.015*"good" + 0.014*"pizza" + 0.014*"time" + 0.013*"food" + 0.013*"great" + 0.011*"well" + 0.009*"order" + 0.007*"price"'),
(8,
'0.032*"food" + 0.026*"good" + 0.026*"place" + 0.015*"great" + 0.009*"service" + 0.008*"time" + 0.006*"price" + 0.006*"meal" + 0.006*"shop" + 0.006*"coffee"'),
(9,
'0.020*"food" + 0.014*"place" + 0.011*"meat" + 0.010*"line" + 0.009*"good" + 0.009*"minute" + 0.008*"time" + 0.008*"chicken" + 0.008*"wing" + 0.007*"hour"')]
------------------------------

PyLDAvis Visualization

My Personal Notes arrow_drop_up