[Paper Review] BERTopic: Neural topic modeling with a class-based TF-IDF procedure

Maarten Grootendorst|arXiv (Cornell University)|Mar 11, 2022

Topic Modeling1,276 citations

TL;DR

BERTopic combines document embeddings, dimensionality reduction, and a class-based TF-IDF to produce coherent topic representations from clustered embeddings, showing competitive performance across datasets and language models.

ABSTRACT

Topic models can be useful tools to discover latent topics in collections of documents. Recent studies have shown the feasibility of approach topic modeling as a clustering task. We present BERTopic, a topic model that extends this process by extracting coherent topic representation through the development of a class-based variation of TF-IDF. More specifically, BERTopic generates document embedding with pre-trained transformer-based language models, clusters these embeddings, and finally, generates topic representations with the class-based TF-IDF procedure. BERTopic generates coherent topics and remains competitive across a variety of benchmarks involving classical models and those that follow the more recent clustering approach of topic modeling.

Motivation & Objective

• Motivate topic modeling with semantic embeddings to overcome bag-of-words limitations.
• Introduce a flexible three-step BERTopic pipeline: embedding, clustering, and topic representation.
• Propose and validate a class-based TF-IDF (c-TF-IDF) for topic word distributions.
• Demonstrate robustness and flexibility across datasets and language models, including dynamic topic modeling capabilities.

Proposed method

• Generate document embeddings using pre-trained language models (SBERT variants).
• Reduce embedding dimensionality with UMAP before clustering with HDBSCAN.
• Create topic representations by applying a class-based TF-IDF (c-TF-IDF) over clusters treated as concatenated documents.
• Iteratively merge c-TF-IDF representations to reach a user-specified number of topics.
• Optionally model dynamic topics by generating global topics and local timestep representations using c-TF-IDF matrices.

Experimental results

Research questions

• RQ1 Can BERTopic produce coherent and diverse topics across diverse datasets using various embedding models?
• RQ2 Does separating embedding/clustering from topic representation improve robustness and flexibility of topic modeling?
• RQ3 Can a class-based TF-IDF provide meaningful topic-word distributions that reflect cluster semantics?
• RQ4 How does BERTopic perform in dynamic/topic-evolution scenarios compared to traditional dynamic topic models?

Key findings

• BERTopic yields high topic coherence across datasets and remains competitive in topic diversity.
• BERTopic performance is stable across different language models, with smaller models offering favorable speed–performance trade-offs.
• In dynamic topic modeling, BERTopic with or without linear evolution assumptions performs well and shows limited impact from the evolution assumption.
• BERTopic can be faster than some neural models when excluding particularly slow baselines, and its speed scales with embedding model choice.
• Separating embedding, clustering, and topic representation enables flexibility such as updating representations without re-clustering.