Introduction
Topic modeling discovers themes in a collection of documents. Latent Dirichlet Allocation (LDA) is a classic topic model.
LDA assumes:
- Each document is a mixture of topics.
- Each topic is a distribution over words.
- Words are generated by first choosing a topic, then choosing a word from that topic.
Intuition
A document about machine learning systems might be:
60% engineering, 25% evaluation, 15% deploymentA topic might assign high probability to words such as:
model, training, metric, feature, deploymentLDA tries to infer these hidden topic and word distributions from observed documents.
Preprocessing
Preprocessing matters:
- Lowercase text.
- Remove boilerplate.
- Tokenize.
- Remove stopwords when appropriate.
- Consider lemmatization.
- Remove extremely rare and extremely common terms.
- Build a document-term matrix.
Bad preprocessing often creates bad topics.
Choosing the Number of Topics
The number of topics is a modeling choice.
Too few topics:
- Themes are mixed together.
Too many topics:
- Topics become fragmented or repetitive.
Use both quantitative metrics and human inspection. Topic coherence can help, but the topics should also be useful for the actual task.
Interpreting Topics
Inspect:
- Top words per topic.
- Representative documents per topic.
- Topic distribution across document groups.
- Stability across random seeds.
Name topics after looking at documents, not only top words.
Limitations
LDA uses a bag-of-words representation, so it ignores word order and much of syntax.
It can struggle with:
- Short documents.
- Noisy text.
- Highly technical vocabulary.
- Topics that depend on phrasing rather than word counts.
For modern semantic discovery, embeddings plus clustering may work better. LDA is still useful when a simple probabilistic model and interpretable word distributions are desired.
Closing
LDA is best understood as a model of document-topic mixtures and topic-word distributions. It is not magic, but it gives a clear framework for exploring large text collections.