[Paper Review] Deep Representation Learning for Clustering of Health Tweets

Oguzhan Gencoglu|arXiv (Cornell University)|Dec 25, 2018

Topic Modeling57 references4 citations

TL;DR

This paper proposes deep convolutional autoencoders (CAEs) for unsupervised representation learning of health-related tweets, enabling superior clustering performance compared to conventional methods like bag-of-words, tf-idf, LDA, and NMF. The method achieves state-of-the-art clustering results using Calinski-Harabasz scores, with an 18% improvement over the next-best baseline, and further enhances performance via an L2-norm constraint on learned representations.

ABSTRACT

Twitter has been a prominent social media platform for mining population-level health data and accurate clustering of health-related tweets into topics is important for extracting relevant health insights. In this work, we propose deep convolutional autoencoders for learning compact representations of health-related tweets, further to be employed in clustering. We compare our method to several conventional tweet representation methods including bag-of-words, term frequency-inverse document frequency, Latent Dirichlet Allocation and Non-negative Matrix Factorization with 3 different clustering algorithms. Our results show that the clustering performance using proposed representation learning scheme significantly outperforms that of conventional methods for all experiments of different number of clusters. In addition, we propose a constraint on the learned representations during the neural network training in order to further enhance the clustering performance. All in all, this study introduces utilization of deep neural network-based architectures, i.e., deep convolutional autoencoders, for learning informative representations of health-related tweets.

Motivation & Objective

• To address the challenge of unsupervised clustering of short, dynamic health-related tweets without labeled data.
• To improve clustering performance by learning compact, informative representations from raw tweet text using deep neural networks.
• To evaluate the effectiveness of deep representation learning against conventional feature extraction techniques in health tweet clustering.
• To investigate the impact of an L2-norm constraint on learned representations to further enhance clustering quality.

Proposed method

• Employ deep convolutional autoencoders (CAEs) to learn low-dimensional, compact representations of raw health-related tweets in an unsupervised manner.
• Apply an L2-norm constraint during training to regularize the latent representations and improve cluster separability.
• Use the learned representations as input to three clustering algorithms: k-means, Ward’s linkage, and spectral clustering.
• Compare performance against conventional methods: bag-of-words, tf-idf, LDA, NMF, and pre-trained word embeddings (word2vec, GloVe, fastText).
• Evaluate clustering quality using the Calinski-Harabasz (CH) score, a normalized internal clustering metric.
• Visualize learned representations using t-SNE and UMAP to assess cluster separability and the impact of the L2 constraint.

Experimental results

Research questions

• RQ1 Can deep convolutional autoencoders learn more effective, compact representations of health-related tweets than conventional feature extraction methods for clustering?
• RQ2 How does the L2-norm constraint on learned representations affect clustering performance across different algorithms?
• RQ3 What is the relative performance of pre-trained word embeddings (e.g., fastText, BERT) compared to traditional feature extraction and deep autoencoders?
• RQ4 Why do high-dimensional representations like raw tf-idf or concatenated word embeddings underperform despite their size?
• RQ5 To what extent do topic modeling methods (LDA, NMF) generalize to short-text tweets in clustering tasks?

Key findings

• The proposed deep convolutional autoencoder with L2-norm constraint achieved the highest Calinski-Harabasz score of 752.4 on 50 clusters, significantly outperforming the next-best method (LDA with tf-idf at 638).
• The L2-norm constrained CAE improved clustering performance across all three clustering algorithms, with the most notable gain in k-means clustering, which relies on L2 distance.
• FastText word embeddings outperformed word2vec and GloVe due to their ability to handle out-of-vocabulary tokens through character n-grams, reducing zero-vector issues in drug name representation.
• Raw tf-idf and bag-of-words features performed poorly, while concatenating word embeddings into high-dimensional vectors led to poor performance due to the curse of dimensionality.
• Topic models (LDA and NMF) underperformed due to the short length of tweets, which limits statistical robustness for topic inference.
• t-SNE and UMAP visualizations confirmed that the L2-constrained CAE produced more separable and well-defined clusters compared to unconstrained training.