[Paper Review] Suicide Phenotyping from Clinical Notes in Safety-Net Psychiatric Hospital Using Multi-Label Classification with Pre-Trained Language Models

Zehan Li, Yan Hu+8 more|PubMed|Sep 27, 2024

Suicide and Self-Harm StudiesPsychology18 references3 citations

TL;DR

This study develops a multi-label suicide phenotyping system using BERT-based models on clinical notes from a safety-net psychiatric hospital. RoBERTa fine-tuned with single multi-label classification achieved the highest performance (accuracy: 0.88, F1: 0.81), demonstrating that domain-specific pretraining and multi-label strategies significantly improve detection of co-occurring suicidal events such as ideation, attempts, exposure, and non-suicidal self-injury.

ABSTRACT

Accurate identification and categorization of suicidal events can yield better suicide precautions, reducing operational burden, and improving care quality in high-acuity psychiatric settings. Pre-trained language models offer promise for identifying suicidality from unstructured clinical narratives. We evaluated the performance of four BERT-based models using two fine-tuning strategies (multiple single-label and single multi-label) for detecting coexisting suicidal events from 500 annotated psychiatric evaluation notes. The notes were labeled for suicidal ideation (SI), suicide attempts (SA), exposure to suicide (ES), and non-suicidal self-injury (NSSI). RoBERTa outperformed other models using binary relevance (acc=0.86, F1=0.78). MentalBERT (F1=0.74) also exceeded BioClinicalBERT (F1=0.72). RoBERTa fine-tuned with a single multi-label classifier further improved performance (acc=0.88, F1=0.81), highlighting that models pre-trained on domain-relevant data and the single multi-label classification strategy enhance efficiency and performance.

Motivation & Objective

• To improve suicide risk detection in high-acuity psychiatric settings by extracting suicidal events from unstructured clinical notes.
• To evaluate the effectiveness of pre-trained language models in identifying co-occurring suicidal phenotypes: suicidal ideation (SI), suicide attempts (SA), exposure to suicide (ES), and non-suicidal self-injury (NSSI).
• To compare multiple fine-tuning strategies—multiple single-label vs. single multi-label—for suicide phenotyping using clinical narratives.
• To assess the impact of model architecture and domain-specific pretraining on phenotyping performance.
• To support clinical decision-making and reduce operational burden through automated, accurate suicide risk phenotyping.

Proposed method

• Fine-tuned four BERT-based models—BERT, BioClinicalBERT, MentalBERT, and RoBERTa—on 500 annotated psychiatric evaluation notes.
• Applied two fine-tuning strategies: multiple single-label classification (one model per suicidal event type) and single multi-label classification (one model predicting all four labels simultaneously).
• Used domain-relevant clinical notes from a safety-net psychiatric hospital to enhance model generalization to mental health contexts.
• Employed standard NLP metrics—accuracy and F1-score—for evaluation across all four suicidal event types.
• Optimized model performance through hyperparameter tuning and attention mechanism utilization in transformer-based architectures.
• Leveraged transfer learning to adapt pre-trained language models to the low-resource, high-stakes domain of suicide phenotyping.

Experimental results

Research questions

• RQ1 Can pre-trained language models effectively detect multiple co-occurring suicidal events from unstructured clinical notes in a psychiatric inpatient setting?
• RQ2 How does the choice of model architecture (e.g., RoBERTa vs. BERT) affect performance in suicide phenotyping tasks?
• RQ3 Does single multi-label classification outperform multiple single-label classification in detecting diverse suicidal phenotypes?
• RQ4 To what extent does domain-specific pretraining improve suicide phenotyping performance compared to general-domain models?
• RQ5 How do different fine-tuning strategies influence the accuracy and F1-score of suicide event detection?

Key findings

• RoBERTa achieved the highest performance using the single multi-label classification strategy, with an accuracy of 0.88 and F1-score of 0.81.
• RoBERTa outperformed other models in the multiple single-label setting, achieving 0.86 accuracy and 0.78 F1-score.
• MentalBERT showed strong performance (0.83 accuracy, 0.74 F1), outperforming BioClinicalBERT (0.82 accuracy, 0.72 F1).
• BioClinicalBERT surpassed the base BERT model (0.80 accuracy, 0.70 F1), indicating the benefit of domain-specific pretraining.
• The single multi-label fine-tuning strategy improved performance over multiple single-label strategies, suggesting better label correlation learning.
• Model optimization, domain-relevant pretraining, and multi-label fine-tuning collectively enhanced suicide phenotyping performance.