[Paper Review] CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling

Zheng Fang, Yulan He+1 more|arXiv (Cornell University)|May 16, 2023

Topic Modeling4 citations

TL;DR

This paper proposes CWTM, a novel neural topic model that leverages contextualized word embeddings from BERT to learn document- and word-level topic vectors without relying on bag-of-words (BOW) representations. By directly mapping BERT's contextualized embeddings to latent topics, CWTM achieves superior topic coherence and robustness to out-of-vocabulary (OOV) words, outperforming LDA and other neural topic models across multiple datasets.

ABSTRACT

Most existing topic models rely on bag-of-words (BOW) representation, which limits their ability to capture word order information and leads to challenges with out-of-vocabulary (OOV) words in new documents. Contextualized word embeddings, however, show superiority in word sense disambiguation and effectively address the OOV issue. In this work, we introduce a novel neural topic model called the Contextlized Word Topic Model (CWTM), which integrates contextualized word embeddings from BERT. The model is capable of learning the topic vector of a document without BOW information. In addition, it can also derive the topic vectors for individual words within a document based on their contextualized word embeddings. Experiments across various datasets show that CWTM generates more coherent and meaningful topics compared to existing topic models, while also accommodating unseen words in newly encountered documents.

Motivation & Objective

• To address the limitations of traditional topic models that rely on bag-of-words (BOW) representations, which ignore word order and struggle with out-of-vocabulary (OOV) words.
• To integrate contextualized word embeddings from BERT into topic modeling to capture context-dependent semantics and improve topic quality.
• To develop a model that learns both document-level and word-level topic vectors directly from contextualized embeddings, avoiding BOW dependency.
• To evaluate the semantic meaningfulness of learned word-topic vectors through downstream NLP tasks like named entity recognition (NER).
• To demonstrate that the model is robust to OOV words in newly encountered documents.

Proposed method

• CWTM maps each word's contextualized embedding from BERT to a latent topic vector using a learnable projection layer.
• Document-level topic vectors are generated via weighted average pooling over the word-topic vectors in the same document.
• The model is trained end-to-end using a reconstruction loss to preserve the relationship between contextualized embeddings and learned topic vectors.
• Word-topic vectors are jointly optimized with document-topic vectors during training, enabling joint inference of topic structure at both word and document levels.
• The model does not require BOW representations as input, relying solely on raw text and BERT's contextualized embeddings.
• Downstream tasks such as NER are evaluated by concatenating CWTM’s learned word-topic vectors with BERT’s last-layer embeddings.

Experimental results

Research questions

• RQ1 Can a neural topic model that bypasses bag-of-words representations and instead uses contextualized word embeddings from BERT produce more coherent and meaningful topics?
• RQ2 To what extent does the model mitigate the out-of-vocabulary (OOV) issue compared to traditional topic models?
• RQ3 Are the learned word-topic vectors semantically meaningful and useful for downstream NLP tasks?
• RQ4 How does the model’s performance in topic coherence and document classification compare to LDA and other neural topic models?
• RQ5 Does the model maintain consistent performance across documents with varying levels of OOV words?

Key findings

• CWTM achieved higher topic coherence scores than LDA and other neural topic models on five benchmark datasets, with an average improvement of 0.093 in C_V and 0.163 in UCI.
• On the OOV robustness test, CWTM showed the smallest accuracy difference (0.021±0.005) between Test1 (≥90% seen words) and Test2 (≤70% seen words), indicating superior handling of unseen words.
• In the CoNLL2003 NER task, combining BERT embeddings with CWTM’s word-topic vectors improved F1 score from 0.814 to 0.828, with a statistically significant t-test result (t = -3.75, p = 0.003).
• The model demonstrated consistent performance across datasets, with CWTM outperforming baseline models in document classification on both seen and OOV-heavy test sets.
• The word-topic vectors learned by CWTM were found to be semantically meaningful, as evidenced by their positive impact on downstream NER performance.
• CWTM successfully eliminated reliance on BOW representations while maintaining or improving topic modeling performance.