[Paper Review] MentSum: A Resource for Exploring Summarization of Mental Health Online Posts

Sajad Sotudeh, Nazli Goharian+1 more|arXiv (Cornell University)|Jun 2, 2022

Mental Health via Writing4 citations

TL;DR

This paper introduces MentSum, a large-scale dataset of over 24,000 mental health-related Reddit posts paired with user-written TL;DR summaries, enabling research in summarization of online mental health content. The study evaluates extractive and abstractive summarization models, finding that abstractive models (e.g., BART) outperform extractive ones, and human evaluation shows system-generated summaries are on average preferred over user-written ones in fluency, informativeness, and conciseness, despite challenges in capturing inferred or nuanced information.

ABSTRACT

Mental health remains a significant challenge of public health worldwide. With increasing popularity of online platforms, many use the platforms to share their mental health conditions, express their feelings, and seek help from the community and counselors. Some of these platforms, such as Reachout, are dedicated forums where the users register to seek help. Others such as Reddit provide subreddits where the users publicly but anonymously post their mental health distress. Although posts are of varying length, it is beneficial to provide a short, but informative summary for fast processing by the counselors. To facilitate research in summarization of mental health online posts, we introduce Mental Health Summarization dataset, MentSum, containing over 24k carefully selected user posts from Reddit, along with their short user-written summary (called TLDR) in English from 43 mental health subreddits. This domain-specific dataset could be of interest not only for generating short summaries on Reddit, but also for generating summaries of posts on the dedicated mental health forums such as Reachout. We further evaluate both extractive and abstractive state-of-the-art summarization baselines in terms of Rouge scores, and finally conduct an in-depth human evaluation study of both user-written and system-generated summaries, highlighting challenges in this research.

Motivation & Objective

• To address the lack of domain-specific datasets for summarizing mental health-related social media posts, particularly for aiding counselors in fast triage.
• To create a high-quality, privacy-protected dataset of Reddit posts from 43 mental health subreddits, each paired with a concise user-generated TL;DR summary.
• To evaluate state-of-the-art extractive and abstractive summarization models on this new dataset using automatic (ROUGE) and human evaluation metrics.
• To conduct a detailed human evaluation comparing system-generated summaries with user-written TL;DRs in terms of fluency, informativeness, and conciseness.
• To identify persistent challenges in summarization models, such as missing inferred content and over-reliance on surface-level cues, to guide future research.

Proposed method

• Curated over 24,000 user posts from 43 public mental health subreddits on Reddit, ensuring diversity across conditions like depression, anxiety, and ADHD.
• Collected user-written TL;DR summaries (short, self-contained summaries) for each post, serving as the gold-standard reference for summarization evaluation.
• Evaluated both extractive (e.g., Lead-3, BERT-based) and abstractive (e.g., BART, T5) summarization models using ROUGE scores to measure automatic performance.
• Conducted a human evaluation on 100 randomly selected post-summary pairs, assessing fluency, informativeness, and conciseness using a pairwise comparison setup.
• Performed error analysis on system-generated summaries to identify failure modes, including omission of inferred content, repetition, and over- or under-summaries.
• Applied a Data Usage Agreement (DUA) to ensure user privacy and ethical data use, despite the public nature of the source posts.

Experimental results

Research questions

• RQ1 How do extractive and abstractive summarization models perform on a new, domain-specific dataset of mental health posts from Reddit?
• RQ2 To what extent do system-generated summaries match or exceed the quality of user-written TL;DRs in terms of fluency, informativeness, and conciseness?
• RQ3 What are the key failure modes of current summarization models when applied to mental health content, particularly regarding inference and nuance?
• RQ4 How does the performance gap between extractive models and the theoretical upper bound (OracleExt) inform future research in extractive summarization for mental health texts?
• RQ5 What characteristics of mental health posts—such as weak lead bias or abstractiveness—pose unique challenges for summarization models?

Key findings

• The abstractive model BART achieved the highest ROUGE scores among all evaluated models, outperforming extractive baselines.
• Despite being human-written, user TL;DRs were outperformed by system-generated summaries in average human preference for fluency, informativeness, and conciseness.
• Extractive models showed a significant performance gap from the OracleExt upper bound, indicating substantial room for improvement in extractive summarization for this domain.
• System-generated summaries often captured all diagnosed mental health conditions (e.g., anxiety, GAD, agoraphobia) more completely than user-written TL;DRs, which sometimes omitted details.
• A key challenge was the model’s inability to infer subtle, contextually implied information—such as emotional burden or social impact—present in some user TL;DRs but absent from the original post.
• In some cases, user-written TL;DRs were less concise than system-generated summaries, suggesting that models can effectively compress information while preserving key points.