[Paper Review] Ex-Twit: Explainable Twitter Mining on Health Data

Tunazzina Islam|arXiv (Cornell University)|May 24, 2019

Topic Modeling4 citations

TL;DR

This paper proposes Ex-Twit, a method that combines topic modeling (LDA, NMF, LSA) with LIME for explainable Twitter mining on health-related data. It achieves 80% test accuracy in topic prediction and provides local, instance-level explanations by highlighting salient words that justify model predictions, enabling trust and interpretability in unsupervised NLP applications.

ABSTRACT

Since most machine learning models provide no explanations for the predictions, their predictions are obscure for the human. The ability to explain a model's prediction has become a necessity in many applications including Twitter mining. In this work, we propose a method called Explainable Twitter Mining (Ex-Twit) combining Topic Modeling and Local Interpretable Model-agnostic Explanation (LIME) to predict the topic and explain the model predictions. We demonstrate the effectiveness of Ex-Twit on Twitter health-related data.

Motivation & Objective

• To address the lack of interpretability in black-box machine learning models used for Twitter mining, especially in health-related contexts.
• To develop a method that not only predicts topics in unstructured Twitter text but also explains the rationale behind each prediction.
• To enable human users to trust and validate model outputs by providing local, word-level explanations using LIME.
• To evaluate the effectiveness of the approach on real-world health-related Twitter data, comparing model predictions with human-annotated topics.
• To demonstrate that explainable AI techniques can be effectively integrated with topic modeling for improved transparency in social media analytics.

Proposed method

• The method employs three topic modeling techniques—Latent Dirichlet Allocation (LDA), Non-negative Matrix Factorization (NMF), and Latent Semantic Analysis (LSA)—to discover underlying topics in health-related tweets.
• The optimal number of topics is selected using coherence score maximization, as visualized in a coherence vs. topic count graph.
• LIME is applied post-topic modeling to generate local, instance-level explanations by perturbing input tweets and identifying the most influential words for each prediction.
• For each tweet, the model assigns a prediction probability and a LIME score to each topic, with green markers indicating words matching topic keywords and red markers indicating mismatches.
• The framework uses mean Kullback-Leibler (KL) divergence to measure the similarity between the local LIME explanation and the global topic model, quantifying explanation fidelity.
• Explanations are visualized to show which words in a tweet most strongly support the predicted topic, enabling human inspection and validation.

Experimental results

Research questions

• RQ1 How can we improve the interpretability of topic modeling predictions in health-related Twitter data?
• RQ2 To what extent do model predictions align with human-annotated topics in real-world Twitter data?
• RQ3 Can LIME provide meaningful, local explanations that clarify why a specific topic was assigned to a given tweet?
• RQ4 How do salient words in a tweet influence the model’s prediction, and can these be visually highlighted for transparency?
• RQ5 What is the performance of the combined LDA-LIME framework in terms of prediction accuracy and explanation quality?

Key findings

• The model achieved 93% accuracy on the training set and 80% accuracy on the test set when comparing predicted topics with human-annotated ground truth.
• For the 4th row of test data, the model predicted Topic 1 with a probability of approximately 0.89 and a LIME score of about 2.4, supported by matching keywords 'diet' and 'gym'.
• In a case where model and human annotations disagreed (6th row), the model predicted Topic 4 with a probability of 0.82, driven by matches with 'fitness', 'wellness', and 'great'—key terms in Topic 4's keyword list.
• The explanation for the misaligned prediction showed that while 'swimming' matched Topic 3 (correct human label), the model favored Topic 4 due to stronger word matches and higher prediction confidence.
• The mean KL divergence was 0.026 for a correctly predicted training example and 0.061 for a misclassified test example, indicating varying explanation fidelity across instances.
• LIME visualizations successfully highlighted both supporting and conflicting words, enabling clear diagnosis of prediction reasoning and discrepancies with human labels.