[Paper Review] Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Health

Shaoxiong Ji, Tianlin Zhang+4 more|arXiv (Cornell University)|Apr 20, 2023

Mental Health via Writing16 citations

TL;DR

The paper trains domain-specific long-context language models MentalXLNet and MentalLongformer on Reddit data and evaluates their effectiveness for mental health classification and long-range text understanding. MentalLongformer generally excels on longer documents.

ABSTRACT

Pretrained language models have been used in various natural language processing applications. In the mental health domain, domain-specific language models are pretrained and released, which facilitates the early detection of mental health conditions. Social posts, e.g., on Reddit, are usually long documents. However, there are no domain-specific pretrained models for long-sequence modeling in the mental health domain. This paper conducts domain-specific continued pretraining to capture the long context for mental health. Specifically, we train and release MentalXLNet and MentalLongformer based on XLNet and Longformer. We evaluate the mental health classification performance and the long-range ability of these two domain-specific pretrained models. Our models are released in HuggingFace.

Motivation & Objective

• Improve mental health classification performance through domain-specific continued pretraining.
• Enable effective modeling of long documents common in mental health data (e.g., Reddit posts).
• Compare long-context models (Longformer, XLNet) with domain-specific baselines on multiple datasets.
• Provide guidance on model choice for short vs. long text in mental health tasks.

Proposed method

• Continue pretraining XLNet and Longformer in a mental-health domain using Reddit-based corpus from relevant subreddits.
• Evaluate on multiple mental health classification datasets with both short and long texts.
• Fine-tune with pooled last-layer representations and an MLP classifier.
• Compare against vanilla pretrained models, domain-specific baselines, and zero-shot ChatGPT baselines.
• Analyze long-range capabilities by varying input sequence lengths and observing performance trends.

Figure 1: Median number of tokens in the datasets

Experimental results

Research questions

• RQ1 Can domain-specific continued pretraining improve mental health classification across diverse datasets?
• RQ2 How do long-context models compare to standard and domain-specific baselines on short vs. long texts?
• RQ3 What is the effect of increasing input sequence length on long-range modeling for mental health tasks?
• RQ4 Which model is preferable for long documents (e.g., CLPsych15) versus shorter posts (e.g., Dreaddit, SAD dataset)?

Key findings

• MentalXLNet and MentalLongformer outperform most baselines on classification tasks, especially for longer sequences like CLPsych15.
• MentalLongformer shows strong performance on long texts and often matches or exceeds Longformer when domain-specific pretraining is applied.
• MentalRoBERTa performs best on shorter, 512-token datasets (Dreaddit), while MentalLongformer is advantageous for longer inputs.
• For long-range analysis, increasing sequence length generally improves recall and F1 in many cases, though fluctuations occur.
• Domain-specific continued pretraining yields consistent gains over non-domain-specific Longformer/XLNet baselines in several datasets.