[Paper Review] CAMS: An Annotated Corpus for Causal Analysis of Mental Health Issues in Social Media Posts

Muskan Garg, Chandni Saxena+5 more|arXiv (Cornell University)|Jul 11, 2022

Mental Health via Writing25 citations

TL;DR

CAMS introduces a publicly available annotated corpus for interpretable causal analysis of mental health in Reddit posts, labeling direct causes into six categories and evaluating classifiers on this task. Logistic Regression bests deep models in accuracy.

ABSTRACT

Research community has witnessed substantial growth in the detection of mental health issues and their associated reasons from analysis of social media. We introduce a new dataset for Causal Analysis of Mental health issues in Social media posts (CAMS). Our contributions for causal analysis are two-fold: causal interpretation and causal categorization. We introduce an annotation schema for this task of causal analysis. We demonstrate the efficacy of our schema on two different datasets: (i) crawling and annotating 3155 Reddit posts and (ii) re-annotating the publicly available SDCNL dataset of 1896 instances for interpretable causal analysis. We further combine these into the CAMS dataset and make this resource publicly available along with associated source code: https://github.com/drmuskangarg/CAMS. We present experimental results of models learned from CAMS dataset and demonstrate that a classic Logistic Regression model outperforms the next best (CNN-LSTM) model by 4.9\% accuracy.

Motivation & Objective

• Define interpretable causal analysis for mental health in social media posts.
• Create an annotation schema and corpus for direct causes of mental health issues.
• Crawl and annotate Reddit data and re-annotate the SDCNL dataset to form CAMS.
• Validate annotations with expert review and Fleiss’ Kappa agreement.
• Evaluate machine learning baselines on CAMS to establish feasibility of causal categorization.

Proposed method

• Propose a six-way causal category scheme: No reason, Bias/Abuse, Jobs/Careers, Medication, Relationship, Alienation.
• Annotate posts with text, cause, and inference, guided by clinical experts.
• Combine crawled Reddit data (3155 final) with re-annotated SDCNL data (SDCNL train/test) to form CAMS (5051 total).
• Use inter-annotator agreement (Fleiss’ Kappa) to validate reliability across datasets.
• Preprocess text with GloVe 100-d embeddings and train multiple classifiers.
• Evaluate LR, SVM, LSTM, CNN, GRU, Bi-LSTM, Bi-GRU, and hybrids on CAMS metrics (F1 per class and accuracy).

Experimental results

Research questions

• RQ1 Can a publicly annotated corpus enable interpretable causal analysis of mental health reasons in social media posts?
• RQ2 What are the six causal categories for mental health causes in posts, and how reliably can humans annotate them?
• RQ3 How well do traditional and deep learning models perform on multi-class causal classification of CAMS?
• RQ4 Does augmenting with crawled data improve class balance and model performance?
• RQ5 What are the common error patterns and limitations in automatic causal classification of mental health causes?

Key findings

• CAMS yields 5051 annotated instances across six cause categories.
• Logistic Regression achieves highest overall accuracy among tested models at 0.5013 on CAMS test data.
• CNN-LSTM (hybrid) is the best deep learning model among those tested with 0.4778 accuracy.
• Deep models generally underperform LR compared to LR on this task, due to class overlap and data characteristics.
• Across models, Alienation and Relationship causes are more challenging, often confused with Bias/Abuse and Medication.
• Inter-annotator agreement on CAMS is 61.28%, indicating substantial but imperfect reliability given task subjectivity.