[Paper Review] Moving Beyond LDA: A Comparison of Unsupervised Topic Modelling Techniques for Qualitative Data Analysis of Online Communities

Amandeep Kaur, James R. Wallace|arXiv (Cornell University)|Dec 19, 2024

Computational and Text Analysis Methods4 citations

TL;DR

This study evaluates BERTopic, a large language model (LLM)-based topic modelling technique, against traditional methods (LDA and NMF) for qualitative analysis of online community data. Researchers found BERTopic superior in generating coherent, detailed, and logically organized topics that reveal nuanced relationships, with 8 out of 12 participants preferring it for deeper insight and actionable findings despite challenges with topic volume.

ABSTRACT

Social media constitutes a rich and influential source of information for qualitative researchers. Although computational techniques like topic modelling assist with managing the volume and diversity of social media content, qualitative researcher's lack of programming expertise creates a significant barrier to their adoption. In this paper we explore how BERTopic, an advanced Large Language Model (LLM)-based topic modelling technique, can support qualitative data analysis of social media. We conducted interviews and hands-on evaluations in which qualitative researchers compared topics from three modelling techniques: LDA, NMF, and BERTopic. BERTopic was favoured by 8 of 12 participants for its ability to provide detailed, coherent clusters for deeper understanding and actionable insights. Participants also prioritised topic relevance, logical organisation, and the capacity to reveal unexpected relationships within the data. Our findings underscore the potential of LLM-based techniques for supporting qualitative analysis.

Motivation & Objective

• To evaluate the usability and effectiveness of BERTopic, an LLM-based topic modelling technique, for qualitative data analysis in online communities.
• To address the barrier faced by qualitative researchers lacking programming expertise in adopting computational topic modelling tools.
• To compare BERTopic with traditional methods (LDA and NMF) in terms of topic quality, interpretability, and researcher preference.
• To identify key design requirements for computational tools that support qualitative researchers in analyzing large-scale social media data.
• To integrate BERTopic into the Computational Thematic Analysis (CTA) toolkit and assess its impact on research workflow and insight generation.

Proposed method

• Integrated BERTopic into the CTA toolkit, leveraging its transformer-based word embeddings and contextual understanding for topic modelling.
• Applied BERTopic, LDA, and NMF to qualitative datasets from online communities, particularly Reddit, to compare topic outputs.
• Used topic coherence and diversity metrics to quantitatively evaluate model performance, with coherence serving as a key evaluation criterion.
• Conducted semi-structured interviews with 12 qualitative researchers to assess usability, topic interpretability, and preference across methods.
• Performed hands-on evaluations where researchers applied the CTA toolkit to their own datasets, comparing topic clusters across models.
• Addressed computational demands by enabling GPU utilization and modifying data filtering and pre-processing pipelines to support BERTopic’s requirements.

Figure 1. Modeling and Sampling Module in the CTA Toolkit with BERTopic Integration: Model Details, a Topic List with keywords, Sample List of entries on the left side, and a Chord Graph for with Topics as Word clouds on the right. This layout facilitates detailed and comprehensive analysis of topic

Experimental results

Research questions

• RQ1 How do qualitative researchers perceive the coherence, relevance, and interpretability of topics generated by BERTopic compared to LDA and NMF?
• RQ2 What are the key challenges qualitative researchers face when using traditional topic modelling tools, particularly regarding programming expertise and data pre-processing?
• RQ3 In what ways does BERTopic support deeper understanding and discovery of unexpected relationships within qualitative data from online communities?
• RQ4 How do researchers prioritize topic organization, logical structure, and actionable insights when selecting a topic modelling method?
• RQ5 What design features—such as visualization, search, or hierarchical structure—are most needed to improve the usability of LLM-based topic models in qualitative research?

Key findings

• Eight out of twelve qualitative researchers preferred BERTopic due to its ability to generate detailed, coherent, and logically organized topic clusters.
• BERTopic outperformed LDA and NMF in topic coherence and diversity metrics, indicating higher-quality and more meaningful topic representations.
• Researchers valued BERTopic’s capacity to reveal unexpected yet significant relationships within the data, enabling deeper and more nuanced analysis.
• Despite its strengths, BERTopic’s large number of topics was perceived as overwhelming by some participants, highlighting a need for better hierarchical visualization.
• LDA and NMF required extensive manual data cleaning due to issues with irrelevant symbols and mathematical characters, whereas BERTopic required less pre-processing.
• The lack of hierarchical visualization in the CTA toolkit was identified as a major limitation, restricting researchers’ ability to explore complex topic structures effectively.

Figure 2. Chord diagram for LDA on the r/MachineLearning dataset, highlighting critiques of irrelevant symbols and the presence of mathematical symbols.