[Paper Review] Machine Learning Approaches for Mental Illness Detection on Social Media: A Systematic Review of Biases and Methodological Challenges

Yuchen Cao, Jianglai Dai+5 more|arXiv (Cornell University)|Oct 21, 2024

Mental Health via WritingPsychology3 citations

TL;DR

This systematic review analyzes machine learning models for detecting depression on social media, identifying critical biases and methodological flaws across the ML lifecycle. It reveals widespread issues including overreliance on English-language Twitter data, non-probability sampling, inconsistent preprocessing, and inadequate handling of class imbalance, urging improved data diversity, standardized practices, and reporting transparency for more reliable and generalizable models.

ABSTRACT

The global increase in mental illness requires innovative detection methods for early intervention. Social media provides a valuable platform to identify mental illness through user-generated content. This systematic review examines machine learning (ML) models for detecting mental illness, with a particular focus on depression, using social media data. It highlights biases and methodological challenges encountered throughout the ML lifecycle. A search of PubMed, IEEE Xplore, and Google Scholar identified 47 relevant studies published after 2010. The Prediction model Risk Of Bias ASsessment Tool (PROBAST) was utilized to assess methodological quality and risk of bias. The review reveals significant biases affecting model reliability and generalizability. A predominant reliance on Twitter (63.8%) and English-language content (over 90%) limits diversity, with most studies focused on users from the United States and Europe. Non-probability sampling (80%) limits representativeness. Only 23% explicitly addressed linguistic nuances like negations, crucial for accurate sentiment analysis. Inconsistent hyperparameter tuning (27.7%) and inadequate data partitioning (17%) risk overfitting. While 74.5% used appropriate evaluation metrics for imbalanced data, others relied on accuracy without addressing class imbalance, potentially skewing results. Reporting transparency varied, often lacking critical methodological details. These findings highlight the need to diversify data sources, standardize preprocessing, ensure consistent model development, address class imbalance, and enhance reporting transparency. By overcoming these challenges, future research can develop more robust and generalizable ML models for depression detection on social media, contributing to improved mental health outcomes globally.

Motivation & Objective

• To identify and analyze biases and methodological challenges in machine learning models for detecting mental illness, particularly depression, using social media data.
• To evaluate the methodological quality and risk of bias across 47 relevant studies published after 2010.
• To assess data diversity, sampling practices, preprocessing techniques, and model evaluation strategies in existing research.
• To highlight gaps in reporting transparency and consistency in model development and hyperparameter tuning.
• To provide actionable recommendations for improving the reliability, generalizability, and ethical deployment of future ML models in mental health detection.

Proposed method

• A systematic literature review was conducted using PubMed, IEEE Xplore, and Google Scholar to identify 47 relevant studies published after 2010.
• The Prediction model Risk Of Bias Assessment Tool (PROBAST) was applied to assess methodological quality and risk of bias in the selected studies.
• Data sources, language distribution, sampling methods, preprocessing practices, hyperparameter tuning, and evaluation metrics were systematically extracted and analyzed.
• The review focused on identifying biases related to data representation, model development, and reporting transparency in mental illness detection using social media content.
• Quantitative analysis of study characteristics was performed to assess trends in data source usage, language, geographic focus, and methodological consistency.
• Findings were synthesized to highlight systemic challenges and to inform best practices for future research.

Experimental results

Research questions

• RQ1 What are the dominant data sources and language distributions in existing machine learning studies for mental illness detection on social media?
• RQ2 To what extent do methodological practices such as sampling, preprocessing, and hyperparameter tuning contribute to bias and reduced model generalizability?
• RQ3 How consistently are evaluation metrics applied, particularly in handling class imbalance common in mental health datasets?
• RQ4 What are the key gaps in reporting transparency and methodological detail across studies?
• RQ5 How do biases in data collection and model development affect the reliability and ethical deployment of mental illness detection systems?

Key findings

• 63.8% of studies relied exclusively on Twitter data, with over 90% of content in English, indicating a significant lack of linguistic and cultural diversity.
• 80% of studies used non-probability sampling, limiting the representativeness and generalizability of model findings.
• Only 23% of studies explicitly addressed linguistic nuances such as negations, which are critical for accurate sentiment and emotion analysis.
• 27.7% of studies reported inconsistent hyperparameter tuning, increasing the risk of overfitting and model instability.
• 17% of studies used inadequate data partitioning strategies, further exacerbating overfitting risks.
• While 74.5% of studies used appropriate evaluation metrics for imbalanced data, many others relied solely on accuracy, which can mislead results in skewed datasets.