[论文解读] Harnessing Large Language Models Over Transformer Models for Detecting Bengali Depressive Social Media Text: A Comprehensive Study

A. K. Azad Chowdhury, Md. Saidur Rahman Sujon+5 more|arXiv (Cornell University)|Jan 14, 2024

Mental Health via Writing被引用 5

一句话总结

本文比较 LLM、PLM 以及深度学习/transformer 模型在孟加拉语抑郁文本检测中的表现，介绍孟加拉语社交媒体抑郁数据集（BSMDD），并报道 DepGPT 在零-shot/少-shot 设置下实现近乎完美的准确率和 F1 值。

ABSTRACT

In an era where the silent struggle of underdiagnosed depression pervades globally, our research delves into the crucial link between mental health and social media. This work focuses on early detection of depression, particularly in extroverted social media users, using LLMs such as GPT 3.5, GPT 4 and our proposed GPT 3.5 fine-tuned model DepGPT, as well as advanced Deep learning models(LSTM, Bi-LSTM, GRU, BiGRU) and Transformer models(BERT, BanglaBERT, SahajBERT, BanglaBERT-Base). The study categorized Reddit and X datasets into "Depressive" and "Non-Depressive" segments, translated into Bengali by native speakers with expertise in mental health, resulting in the creation of the Bengali Social Media Depressive Dataset (BSMDD). Our work provides full architecture details for each model and a methodical way to assess their performance in Bengali depressive text categorization using zero-shot and few-shot learning techniques. Our work demonstrates the superiority of SahajBERT and Bi-LSTM with FastText embeddings in their respective domains also tackles explainability issues with transformer models and emphasizes the effectiveness of LLMs, especially DepGPT, demonstrating flexibility and competence in a range of learning contexts. According to the experiment results, the proposed model, DepGPT, outperformed not only Alpaca Lora 7B in zero-shot and few-shot scenarios but also every other model, achieving a near-perfect accuracy of 0.9796 and an F1-score of 0.9804, high recall, and exceptional precision. Although competitive, GPT-3.5 Turbo and Alpaca Lora 7B show relatively poorer effectiveness in zero-shot and few-shot situations. The work emphasizes the effectiveness and flexibility of LLMs in a variety of linguistic circumstances, providing insightful information about the complex field of depression detection models.

研究动机与目标

• 调查多样化 NLP 模型（深度学习、变换器/ transformers 和 LLMs）在孟加拉语抑郁文本检测中的有效性。
• 从 Reddit 和 X 的翻译中创建并验证一个孟加拉语抑郁文本数据集（BSMDD），并确保高注释质量。
• 评估包括 DepGPT、GPT-4、GPT-3.5 以及 Alpaca LoRA 7B 在零-shot 与少-shot 学习上的性能。

提出的方法

• 将来自 Reddit 和 X 的孟加拉语抑郁内容翻译并注释到孟加拉语社交媒体抑郁数据集（BSMDD）。
• 对文本进行预处理，去噪、去重并标准化以用于模型训练。
• 使用 Word2vec、GloVe 和 FastText 嵌入对深度学习模型（LSTM、BiLSTM、GRU、BiGRU）进行评估。
• 微调大语言模型（GPT-3.5 Base、GPT-3.5 Turbo、GPT-4、DepGPT、Alpaca LoRA 7B）并与 PLMs（BERT Multilingual、BanglaBERT、sahajBERT、Bangla BERT Base）进行比较。
• 设计用于对 LLMs 进行零-shot/少-shot 评估的提示词和系统提示。
• 报告准确率、精确度、召回率和 F1 作为主要指标。

实验结果

研究问题

• RQ1 哪一类模型（深度学习、PLM 转换器，还是 LLMs）在孟加拉语抑郁文本检测中表现最佳？
• RQ2 零-shot 和少-shot 提示如何影响孟加拉语抑郁文本分类的准确率和 F1 值？
• RQ3 在孟加拉语抑郁数据集上，DepGPT 相对于 GPT-3.5 Turbo、GPT-4 和 Alpaca LoRA 7B 的对比性能如何？

主要发现

• DepGPT 在零-shot/少-shot 设置下实现了近乎完美的准确率 0.9796 和 F1-score 0.9804。
• SahajBERT 与使用 FastText 嵌入的 Bi-LSTM 在各自领域表现出色。
• GPT-3.5 Turbo 与 Alpaca LoRA 7B 竞争激烈，但在零-shot/少-shot 场景普遍不如 DepGPT。
• 该研究提供了在 DL、PLMs 和 LLMs 之间评估孟加拉语抑郁文本的架构细节和方法，同时解决了变换器的可解释性问题。