[Paper Review] Mental Illness Classification on Social Media Texts using Deep Learning and Transfer Learning

Iqra Ameer, Muhammad Arif+3 more|arXiv (Cornell University)|Jul 3, 2022

Mental Health via Writing28 citations

TL;DR

The paper compares traditional ML, deep learning, and transfer learning models for multi-class mental illness detection on Reddit posts, finding RoBERTa as the best transfer-learning approach with 0.83 accuracy. Bi-LSTM also performs well among DL methods.

ABSTRACT

Given the current social distance restrictions across the world, most individuals now use social media as their major medium of communication. Millions of people suffering from mental diseases have been isolated due to this, and they are unable to get help in person. They have become more reliant on online venues to express themselves and seek advice on dealing with their mental disorders. According to the World health organization (WHO), approximately 450 million people are affected. Mental illnesses, such as depression, anxiety, etc., are immensely common and have affected an individuals' physical health. Recently Artificial Intelligence (AI) methods have been presented to help mental health providers, including psychiatrists and psychologists, in decision making based on patients' authentic information (e.g., medical records, behavioral data, social media utilization, etc.). AI innovations have demonstrated predominant execution in numerous real-world applications broadening from computer vision to healthcare. This study analyzes unstructured user data on the Reddit platform and classifies five common mental illnesses: depression, anxiety, bipolar disorder, ADHD, and PTSD. We trained traditional machine learning, deep learning, and transfer learning multi-class models to detect mental disorders of individuals. This effort will benefit the public health system by automating the detection process and informing appropriate authorities about people who require emergency assistance.

Motivation & Objective

• Motivate automated mental health screening using social media data due to increased online communication during COVID-19 and social distancing.
• Investigate the feasibility of detecting five mental illnesses and a none class from Reddit posts.
• Compare traditional machine learning, deep learning, and transfer learning approaches on a standardized Reddit dataset.
• Assess class-wise performance and analyze challenges in detecting depression and anxiety from short posts.

Proposed method

• Apply four traditional ML classifiers with TF-IDF word n-grams up to length 3.
• Evaluate deep learning models including GRU, Bi-GRU, CNN, LSTM, and Bi-LSTM.
• Fine-tune pre-trained transformer models BERT, XLNet, and RoBERTa on the dataset.
• Use a pre-trained RoBERTa base model as the primary transfer learning approach for multi-class classification.
• Use the provided Reddit dataset of 16,930 posts with train/dev/test splits for evaluation.

Figure 1: Mental Illness Dataset Statistics

Experimental results

Research questions

• RQ1 Can traditional ML methods with n-gram features achieve competitive accuracy on Reddit mental illness classification?
• RQ2 Do deep learning models outperform traditional ML on this task?
• RQ3 Which transfer-learning models (BERT, XLNet, RoBERTa) yield the best performance for multi-class mental illness detection on Reddit?
• RQ4 How well do models distinguish the none class and what errors are common among illness classes?

Key findings

• RoBERTa achieves the highest overall accuracy of 0.83 and F1-score of 0.83 among tested models.
• XLNet is close to RoBERTa, with an accuracy of 0.79 and F1-score of 0.80.
• Bi-LSTM performs best among deep learning models with an accuracy of 0.78 and F1-score of 0.79.
• Traditional ML with word 1–3-grams achieves 0.78 accuracy and 0.67 F1-score, showing competitive performance.
• ROBOTa class-wise results show very high performance for none (F1 0.98) and notable variation across illness classes, with ADHD and PTSD achieving higher F1 than depression and anxiety.
• RoBERTa can correctly identify none-class posts with high precision and recall, suggesting low false positives for non-illness content.

Figure 2: RoBERTa confusion matrix