Exploring Topic Modeling Techniques: LDA, NMF, BERT, Multi-layer Prediction
From news websites, we often see articles categorized into different columns such as Business, Technology, Entertainment, and Sports. The process of assigning these articles to categories is known as Topic Modeling (or Topic Tagging). This is an important product feature that allows users to quickly identify content matching their interests
Introduction
Topic modeling algorithms have two types: extractive and predictive.
Extractive Topic Modeling does not have a predefined set of topics, topics come from texts. We develop rules to define what are important words from sentences (could be word frequency), and slightly adjust them into topics. In the below sentence, the extracted words could be financial, currency, United States. So this sentence may appear under Finance, Currency, and U.S. categories.
"Our expectation is that financial institutions and digital asset companies and others in the virtual currency ecosystem take steps to prevent terrorists from being able to access resources. If they do not act to prevent illicit financial flows, the United States and our partners will," Adeyemo added.
The most important characteristic of extractive topic modeling is that one sentence can have more than one topic. This determines the application of this approach is more on things that require less strict categories and fine with the same sentence appearing under multiple tags.
In contrast, Predictive Topic Modeling has a predefined set of topics, and we try to classify sentences into the most likely category, so each sentence can only have one topic. If our topic taxonomy is Business, Finance, Technology, Entertainment, Health, the sample sentence is most likely to be classified as Finance. Though it’s also reasonable to classify it as business and technology, the probabilistic model only gives one highest probability.
Predictive modeling is particularly useful when we want to categorize things into a single group, such as predicting movie genres or classifying content into specific ESG (Environmental, Social, and Governance) categories.
Let’s compare the two methodologies side by side.
| Extractive Topic Modeling | Predictive Topic Modeling |
Predefined Topics | No | Yes |
Topic Amount | Fixed, can be large. Experiment to decide the optimal number. | Fixed, usually small to ensure good performance. |
Algorithm | Unsupervised | Supervised |
Training | No | Need dataset for each topic |
Use Cases | Product Reviews, Online Campaign Investigation | News Columns, Movie Classification, ESG Rating |
Extractive Topic Modeling
Latent Dirichlet Allocation (LDA)
Latent Dirichlet Allocation is the most widely used method of extractive topic modeling. The core of this method is a generative statistical model. It belongs to Bayesian statistics as the word latent suggests: we don’t explicitly know the distribution of topics, but through learning the patterns of words in each article, we infer the distribution iteratively. This iterative process is a bit similar to k-means clustering. Now let’s walk through it step by step.
Theoretical preparation
We assume there are a certain number of topics across all documents. This number is chosen by us by our prior knowledge of the collection or can be just random to start with. Each document in the collection is treated as a weighted combination of all topics.
For example, one article could be 60% politics + 20% business + 10% economics + 10% celebrity. Moving deeper, each topic is treated as a collection of words, like the topic politics can include the words ‘government’, ‘policy’, ‘election’, ‘democracy’, etc. Each word has a certain probability of belonging to this topic distribution.
Text Preprocessing
All text-related tasks start with text preprocessing. To achieve the state where an article is a collection of important words, we need to do a series of operations.
We may start with tokenization, to break an article into pieces of words. Then remove stopwords, numbers and punctuations, etc. to only keep words with distinct meanings. After that, we can apply lowercase and stemming or lemmatization to restore words to their root form. These steps are not fixed, we can try different combinations depending on the text.
The LDA Process
Step #1: Random Assignment
The process starts by randomly assigning each of the words in a document to the topics we predefined. One thing to clarify here is we only have a predefined number of topics, not the name of the topic, by far the topic is just a collection of words, we don’t know whether it’s politics, business or finance. This initial assignment generates two things, topic distributions across all documents, and word distributions within topics. But they do not have any practical meaning yet.
Step #2: Iterative Update
To make these distributions actually mean something, we iteratively go through all documents, checking each word and reassigning it to a better topic, considering:
What's the probability this word belongs to this topic?
What's the probability this document is generated by this topic?
After each iteration, the word assignment of the topic improves to some degree.
Step #3: Convergence
After some number of iterations, the word assignment will no longer change. We reached the convergence and now the distributions are meaningful. The final result is, each article is associated with a distribution of topics, like 50% topic1, 40% topic2, 30% topic3, the three topics we assign the document.
To understand what each topic is about, we look at the collection of words it contains. We can assign the topic a name if we want, but it’s usually very challenging as topics often appear as a mixture of various things, making it hard to give them a single, definitive label.
Now we understand the entire workflow of LDA topic modeling, let’s implement the algorithm in Python.
Implementation
We use nltk for text preprocessing, and Gensim to convert a collection of raw text documents into a format suitable for LDA and model building.
# import packages
import nltk
from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer
import string
from gensim import corpora, models
import gensim
nltk.download('stopwords')
nltk.download('wordnet')
# Prepare sample documents
docs = [
"Sugar is bad to consume. My sister likes to have sugar, but not my father.",
"My father spends a lot of time driving my sister around to dance practice.",
"Doctors suggest that driving may cause increased stress and blood pressure.",
"Sometimes I feel pressure to perform well at school, but my father never seems to drive my sister to do better.",
"Health experts say that Sugar is not good for your lifestyle."
]
# Preprocess documents
stop = set(stopwords.words('english'))
exclude = set(string.punctuation)
lemma = WordNetLemmatizer()
def clean(doc):
stop_free = " ".join([word for word in doc.lower().split() if word not in stop])
punc_free = ''.join(ch for ch in stop_free if ch not in exclude)
normalized = " ".join(lemma.lemmatize(word) for word in punc_free.split())
return normalized
doc_clean = [clean(doc).split() for doc in docs]
# Create a dictionary from the cleaned data
dictionary = corpora.Dictionary(doc_clean)
# Convert dictionary to Document-Term Matrix
doc_term_matrix = [dictionary.doc2bow(doc) for doc in doc_clean]Let’s check the matrix we build.
[[(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 2)],
[(2, 1), (4, 1), (6, 1), (7, 1), (8, 1), (9, 1), (10, 1), (11, 1), (12, 1)],
[(8, 1), (13, 1), (14, 1), (15, 1), (16, 1), (17, 1), (18, 1), (19, 1), (20, 1)],
...]Each sublist represents a document. The first element of the tuple is word ID, the second element is frequency. So (0,1) from doc_term_matrix[0][0] means word 0 appears 2 times in the first document of this collection.
The next step is to train the LDA model. Here we specify the number of topics to be 3 and iterate 50 times.
# Creating and Training the LDA model
ldamodel = gensim.models.ldamodel.LdaModel(doc_term_matrix, num_topics=3, id2word = dictionary, passes=50)
# Print the topics
for idx, topic in ldamodel.print_topics(-1):
print("Topic: {} \nWords: {}".format(idx, topic))The coefficient before each word is the likelihood of the word belonging to the distribution of this topic.
Topic: 0
Words: 0.029*"sugar" + 0.029*"driving" + 0.029*"pressure" + 0.029*"bad" + 0.029*"consume" + 0.029*"like" + 0.029*"time" + 0.029*"lot" + 0.029*"spends" + 0.029*"practice"
Topic: 1
Words: 0.084*"father" + 0.084*"sister" + 0.059*"sugar" + 0.034*"feel" + 0.034*"better" + 0.034*"drive" + 0.034*"well" + 0.034*"school" + 0.034*"never" + 0.034*"perform"
Topic: 2
Words: 0.050*"driving" + 0.050*"pressure" + 0.050*"stress" + 0.050*"suggest" + 0.050*"doctor" + 0.050*"increased" + 0.050*"blood" + 0.050*"cause" + 0.050*"may" + 0.050*"health"Now let’s visualize the results.
from wordcloud import WordCloud
import matplotlib.pyplot as plt
# Generating word cloud for each topic
for t in range(ldamodel.num_topics):
plt.figure()
plt.imshow(WordCloud().fit_words(dict(ldamodel.show_topic(t, 200))))
plt.axis("off")
plt.title("Topic #" + str(t))
plt.show()Word clouds show the words in each topic. Looking at the content, we can see it’s indeed hard to give each topic a specific name.
Let’s also check the distribution of words and distance of topics.
import pyLDAvis.gensim_models
# Visualize the topics
pyLDAvis.enable_notebook()
vis = pyLDAvis.gensim_models.prepare(ldamodel, doc_term_matrix, dictionary)
pyLDAvis.display(vis)
Lastly, let’s check the distribution of topics.
doc_topics = [ldamodel.get_document_topics(item) for item in doc_term_matrix]
for doc_num, topics in enumerate(doc_topics):
print(f"Document {doc_num}:")
for topic, prob in topics:
print(f"Topic {topic}, Probability: {prob:.2%}")
print("\n")
Document 0:
Topic 0, Probability: 4.23%
Topic 1, Probability: 91.45%
Topic 2, Probability: 4.32%
Document 1:
Topic 0, Probability: 3.40%
Topic 1, Probability: 93.15%
Topic 2, Probability: 3.44% Non-negative Matrix Factorization (NMF)
The idea of Non-negative Matrix Factorization is to first convert a collection of documents to a large matrix that only contains positive numbers and zero (non-negative), and then factorize it into two smaller matrices where one is document x topic and the other is topic x word.
Similar to LDA, we also need to predefine the number of topics, this will determine the size of the matrix. To construct this matrix, we can follow a similar text preprocessing process in LDA.
Matrix Factorization
V is the Original Matrix, where each row represents a document and each column represents a word. A row from this matrix records the normalized occurrence of words in the document.
W is the Feature Matrix, where each column represents a topic, so the number of columns is predefined by us. Each row represents a document. A row in this matrix tells us what are the possible topics for this document.
H is the Coefficient Matrix. A row in this matrix shows what words belong to the topic. Unlike LDA, the value of coefficients are not probabilities.
Optimization
The ideal case is to factor W into the multiplication of W and H, but in practice we hardly can do it. Alternatively, we do V ≈ W x H. NMF iteratively updates the values in W and H to reconstruct V. The process stops when the reconstruction error (difference between V and the product of W and H, ||V-WH||) is minimized or after a set number of iterations.
Implementation
We use the same text preprocessing steps. To load NMF model, we can directly use sklearn.
from sklearn.decomposition import NMF
from sklearn.feature_extraction.text import TfidfVectorizer
# Convert sparse gensim matrix to dense matrix for NMF
dense_doc_term_matrix = gensim.matutils.corpus2dense(doc_term_matrix, num_terms=len(dictionary)).T
# Apply NMF
num_topics = 3
nmf_model = NMF(n_components=num_topics, random_state=42)
nmf_model.fit(dense_doc_term_matrix)
# Extract the word-topic matrix (H Matrix)
nmf_W = nmf_model.components_
# Display topics
words = list(dictionary.values()) # List of words in the dictionary
for i, topic in enumerate(nmf_W):
top_word_indices = topic.argsort()[-10:]
top_words = [words[j] for j in top_word_indices]
print(f"Topic {i+1}: {' '.join(top_words)}")Topic 1: drive never perform school seems sometimes well feel sister father
Topic 2: practice lot dance around like consume bad sister father sugar
Topic 3: dance pressure doctor cause suggest stress blood increased may drivingSimilar to LDA, the resulting topics are sometimes hard to interpret.
Predictive Topic Modeling
Unlike LDA and NMF which discover topics without prior labels, predictive topic modeling uses supervised learning to assign text to one of the predefined categories based on features learned from a labeled training set.
Implementation
One way to do predictive modeling is to use pre-trained models from HuggingFace and finetune them using a dataset labeled with topics. Below is a skeleton of the implementation.
from transformers import AutoModelForSequenceClassification, AutoTokenizer, TrainingArguments, Trainer
import pandas as pd
from sklearn.model_selection import train_test_split
from datasets import Dataset
# Replace these placeholders with your actual file paths and model names
local_dataset_path = "<LOCAL_DATASET_PATH>"
model_name = "<MODEL_NAME>"
tokenizer_name = "<TOKENIZER_NAME>"
# Load dataset
df = pd.read_csv(local_dataset_path)
train_df, test_df = train_test_split(df, test_size=0.2)
# Convert DataFrame to Hugging Face Dataset
train_dataset = Dataset.from_pandas(train_df)
test_dataset = Dataset.from_pandas(test_df)
# Tokenization
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
def tokenize_function(examples):
return tokenizer(examples["text"], padding="max_length", truncation=True)
tokenized_train_dataset = train_dataset.map(tokenize_function, batched=True)
tokenized_test_dataset = test_dataset.map(tokenize_function, batched=True)
# Load and train model
num_labels = df['label'].nunique()
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=num_labels)
training_args = TrainingArguments(
output_dir="test_trainer",
evaluation_strategy="epoch",
num_train_epochs=2
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_train_dataset,
eval_dataset=tokenized_test_dataset
)
trainer.train()
# Prediction
sentences = ["Sample sentence 1.", "Sample sentence 2."]
predict_inputs = tokenizer(sentences, padding=True, truncation=True, max_length=512, return_tensors="pt")
outputs = trainer.predict(predict_inputs)
predictions = outputs.predictions.argmax(-1)
labels = [model.config.id2label[pred] for pred in predictions]
print(labels)Multi-layer Topic Modeling
In practice, a single-layer predictive model may not be enough. For example, we know an article is about currency, but we are also curious about what kind of currency it’s talking about, is it crypto or US dollar, etc. A multi-layer topic taxonomy is required for this task. A good candidate to start with is IPTC topics (International Press Telecommunications Council) which is widely used by news agencies and publishers.
In multi-layer topic modeling, we can use a hierarchical structure to assign more than one predefined topics from different layers to an article sequentially, combining the advantage of both extractive and predictive topic models.
Text: For Ford, the breakup comes with a twist of irony: The company helped Rivian get off the ground by investing in the electric vehicle start-up in 2019.
We are only scratching the surface...
If you're interested in learning more about how to implement topic modelling to your use case, feel free to reach out to us by leaving your contact information!
View certificate