[Paper Review] Transgender Community Sentiment Analysis from Social Media Data: A Natural Language Processing Approach

Mengzhe Li, Yudan Wang+2 more|arXiv (Cornell University)|Oct 25, 2020

Topic Modeling13 references4 citations

TL;DR

This study develops a natural language processing (NLP) framework for sentiment analysis of transgender individuals' social media posts using a manually annotated dataset of 300 Reddit comments. It evaluates multiple machine learning and deep learning models, with the LSTM model achieving the highest performance (accuracy: 0.852, AUC: 0.876), demonstrating strong potential for early mental health intervention through automated sentiment detection.

ABSTRACT

Transgender community is experiencing a huge disparity in mental health conditions compared with the general population. Interpreting the social medial data posted by transgender people may help us understand the sentiments of these sexual minority groups better and apply early interventions. In this study, we manually categorize 300 social media comments posted by transgender people to the sentiment of negative, positive, and neutral. 5 machine learning algorithms and 2 deep neural networks are adopted to build sentiment analysis classifiers based on the annotated data. Results show that our annotations are reliable with a high Cohen's Kappa score over 0.8 across all three classes. LSTM model yields an optimal performance of accuracy over 0.85 and AUC of 0.876. Our next step will focus on using advanced natural language processing algorithms on a larger annotated dataset.

Motivation & Objective

• To manually annotate 300 Reddit comments from the transgender community into positive, negative, or neutral sentiment categories.
• To evaluate the performance of multiple machine learning and deep learning models on classifying transgender community sentiment from social media text.
• To develop an automated sentiment classifier capable of detecting mental health-related distress in transgender individuals for early intervention.
• To establish a reliable, high-agreement dataset for future NLP research on transgender mental health.

Proposed method

• A dataset of 300 Reddit comments from /r/asktransgender was manually annotated by two raters with a third resolving disagreements, achieving a Cohen’s Kappa > 0.8 across all classes.
• Five traditional machine learning models (Naive Bayes, Logistic Regression, Random Forest, SVM, K-Nearest Neighbors) and two deep learning models (CNN and LSTM) were trained on the annotated data.
• Word2Vec embeddings pre-trained on large-scale social media data were used to represent text inputs for deep learning models.
• The LSTM model used three stacked layers with 128, 64, and 32 units, ReLU activation, dropout (0.2), and global max pooling; Adam optimizer and binary cross-entropy loss were applied.
• Model evaluation used 20% held-out test set for accuracy and macro-averaged AUC; 5-fold cross-validation was used for traditional models, and 10% of training data served as development set for hyperparameter tuning.
• All models were implemented in Python 3.6 using Keras for deep learning and scikit-learn for classical models, with training accelerated on a single GPU.

Experimental results

Research questions

• RQ1 Can a manually annotated dataset of transgender social media comments achieve high inter-annotator agreement for sentiment classification?
• RQ2 Which machine learning and deep learning models perform best on classifying sentiment in transgender community social media content?
• RQ3 Can an automated sentiment classifier detect negative sentiment in transgender individuals’ online expressions with high accuracy and AUC?
• RQ4 How do key words such as 'dysphoria' or 'question' contribute to sentiment classification in this context?

Key findings

• The manual annotation process achieved a Cohen’s Kappa score above 0.8 across all three sentiment classes, indicating strong inter-rater reliability.
• The LSTM model achieved the highest accuracy of 0.852 and the best AUC of 0.876, outperforming all other models tested.
• The CNN model achieved the second-highest accuracy (0.861), slightly outperforming the LSTM in accuracy but not in AUC.
• All deep learning models (CNN and LSTM) outperformed all traditional machine learning classifiers in both accuracy and AUC.
• The word cloud visualization identified 'trans', 'transgender', 'dysphoria', 'question', and 'wondering' as the most influential words for sentiment classification.
• The study demonstrates that automated sentiment analysis on transgender social media data can support early detection of mental health distress and inform timely interventions.