[Paper Review] Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling

Yida Mu, Chun Dong+2 more|arXiv (Cornell University)|Mar 24, 2024

Computational and Text Analysis Methods11 citations

TL;DR

This paper assesses whether large language models (LLMs) can directly extract and summarize topics from corpora, comparing GPT-3.5 and LLaMA-2-7B against LDA and BERTopic, with prompt engineering, seeds, and summarisation to produce human-interpretable topics and evaluation metrics.

ABSTRACT

Topic modelling, as a well-established unsupervised technique, has found extensive use in automatically detecting significant topics within a corpus of documents. However, classic topic modelling approaches (e.g., LDA) have certain drawbacks, such as the lack of semantic understanding and the presence of overlapping topics. In this work, we investigate the untapped potential of large language models (LLMs) as an alternative for uncovering the underlying topics within extensive text corpora. To this end, we introduce a framework that prompts LLMs to generate topics from a given set of documents and establish evaluation protocols to assess the clustering efficacy of LLMs. Our findings indicate that LLMs with appropriate prompts can stand out as a viable alternative, capable of generating relevant topic titles and adhering to human guidelines to refine and merge topics. Through in-depth experiments and evaluation, we summarise the advantages and constraints of employing LLMs in topic extraction.

Motivation & Objective

• Motivate the use of LLMs as a plug-and-play alternative to classical topic modelling for open-domain and domain-specific corpora.
• Investigate prompting techniques, including constraints and seeds, to guide LLMs toward granular, human-interpretable topics.
• Develop evaluation protocols to assess topic quality, granularity, and alignment with human expectations.
• Demonstrate a case study on temporal analysis of vaccine hesitancy to show dynamic topic extraction and summarisation capabilities.
• Compare LLM-driven topic extraction with traditional approaches (LDA, BERTopic) across datasets.

Proposed method

• Use GPT-3.5 and LLaMA-2-7B as base LLMs for topic extraction from document sets.
• Prompt with system and user messages to generate topic lists, exploring out-of-box, constraints, and manual post-processing.
• Introduce seed topics to steer granularity and reduce topic overlap.
• Apply a summarisation step to merge and distill topics into a final Top-N list with explanations.
• Propose evaluation metrics including topic distance (Jaccard) and semantic granularity (BERT cosine similarity) and recall/precision against seed topics.
• Provide a case study on temporal analysis of Twitter data about COVID-19 vaccine hesitancy to illustrate dynamics over time.

Figure 1: Example prompts for LLaMA. Text enclosed by the special tokens ‘ $\ll$ SYS $\gg$ ’ is designated as a system prompt.

Experimental results

Research questions

• RQ1 Can LLMs generate meaningful, human-interpretable topics directly from raw text without classical topic modelling pre-processing?
• RQ2 How do prompting strategies (basic prompts, seeds, summarisation) affect topic quality and granularity across open-domain and domain-specific corpora?
• RQ3 What evaluation metrics best capture the quality, distinctness, and granularity of LLM-generated topics?
• RQ4 Can LLMs adapt to evolving topics in dynamic corpora (e.g., temporal Twitter data) without re-running traditional models?

Key findings

• LLMs can generate topics that are interpretable and align with human expectations when guided by appropriate prompts and constraints.
• Seed topics consistently improve topic extraction quality and granularity across datasets and LLMs.
• A final Top-N summarisation step yields a compact, representative set of topics that covers most original categories and provides explanations.
• Evaluation metrics show that final topic lists are largely distinct (low overlap) and exhibit increasing semantic granularity with larger Top-N sets.
• In a temporal vaccine hesitancy case study, LLMs demonstrate capacity to extract evolving topics over time and produce descriptive analyses (e.g., via GPT-4 descriptions).

Figure 2: Two examples of final topics summarised by LLMs, namely ‘Technology & Computers’ (20NG) and ‘Trust & Mistrust’ (Vaccine dataset).