[Paper Review] MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare

Shaoxiong Ji, Tianlin Zhang+4 more|arXiv (Cornell University)|Oct 29, 2021

Mental Health via WritingPsychology38 references142 citations

TL;DR

The paper trains and releases MentalBERT and MentalRoBERTa on mental health Reddit data, then evaluates them on multiple mental health detection tasks, showing domain-specific pretraining improves performance.

ABSTRACT

Mental health is a critical issue in modern society, and mental disorders could sometimes turn to suicidal ideation without adequate treatment. Early detection of mental disorders and suicidal ideation from social content provides a potential way for effective social intervention. Recent advances in pretrained contextualized language representations have promoted the development of several domain-specific pretrained models and facilitated several downstream applications. However, there are no existing pretrained language models for mental healthcare. This paper trains and release two pretrained masked language models, i.e., MentalBERT and MentalRoBERTa, to benefit machine learning for the mental healthcare research community. Besides, we evaluate our trained domain-specific models and several variants of pretrained language models on several mental disorder detection benchmarks and demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks.

Motivation & Objective

• Motivate the need for domain-specific language models in mental healthcare.
• Provide publicly available pretrained models to the research community.
• Demonstrate that continued pretraining on mental health data improves downstream classification.
• Evaluate models on diverse mental health detection benchmarks across platforms.

Proposed method

• Adapt BERT and RoBERTa starting from their general-domain checkpoints via continued pretraining on a mental health Reddit corpus.
• Use masked language modeling with dynamic masking for MentalRoBERTa and base BERT for MentalBERT.
• Fine-tune the pretrained models on downstream tasks using the [CLS] embedding and an MLP classifier.
• Evaluate models on multiple mental health datasets spanning depression, anxiety, stress, and suicidal ideation.

Experimental results

Research questions

• RQ1 Do domain-specific pretrained models for mental health outperform general-domain pretrained models on mental health detection tasks?
• RQ2 Is continued pretraining on a mental health corpus more effective than domain-adaptive pretraining from other domains (e.g., biomedical/clinical) for mental health tasks?
• RQ3 How do MentalBERT and MentalRoBERTa perform across diverse datasets and platforms (Reddit, Twitter, SMS-like data) for various mental disorders?

Key findings

• MentalRoBERTa often achieves the best results on depression datasets (eRisk T1 and CLPsych) and on several other disorder datasets (T-SID, SWMH, SAD, Dreaddit).
• MentalBERT is competitive and achieves the best F1 on the Depression_Reddit dataset and the UMD dataset’s F1 in some settings.
• Domain-specific pretraining on mental health data generally yields better performance than general-domain models and sometimes outperforms biomedical/clinical-domain pretraining.
• Continued pretraining on the mental health domain improves downstream classification performance across multiple tasks.