[Paper Review] belabBERT: a Dutch RoBERTa-based language model applied to psychiatric classification

Joppe Wouts, Janna N. de Boer+4 more|arXiv (Cornell University)|Jun 2, 2021

Mental Health via Writing4 citations

TL;DR

This paper introduces belabBERT, a Dutch RoBERTa-based language model fine-tuned on a 32 GB Dutch web corpus to improve psychiatric classification. It outperforms both the existing RobBERT model and audio-only classification methods in identifying psychiatric disorders from clinical text, demonstrating the effectiveness of large-scale, contextualized NLP for low-resource languages like Dutch.

ABSTRACT

Natural language processing (NLP) is becoming an important means for automatic recognition of human traits and states, such as intoxication, presence of psychiatric disorders, presence of airway disorders and states of stress. Such applications have the potential to be an important pillar for online help lines, and may gradually be introduced into eHealth modules. However, NLP is language specific and for languages such as Dutch, NLP models are scarce. As a result, recent Dutch NLP models have a low capture of long range semantic dependencies over sentences. To overcome this, here we present belabBERT, a new Dutch language model extending the RoBERTa architecture. belabBERT is trained on a large Dutch corpus (+32 GB) of web crawled texts. We applied belabBERT to the classification of psychiatric illnesses. First, we evaluated the strength of text-based classification using belabBERT, and compared the results to the existing RobBERT model. Then, we compared the performance of belabBERT to audio classification for psychiatric disorders. Finally, a brief exploration was performed, extending the framework to a hybrid text- and audio-based classification. Our results show that belabBERT outperformed the current best text classification network for Dutch, RobBERT. belabBERT also outperformed classification based on audio alone.

Motivation & Objective

• To address the scarcity of high-performing NLP models for Dutch, particularly in capturing long-range semantic dependencies.
• To develop a robust, large-scale language representation model tailored for psychiatric text classification in Dutch.
• To compare the performance of text-based classification using belabBERT against existing models like RobBERT and audio-based classification methods.
• To explore the potential of hybrid text- and audio-based classification for improved psychiatric disorder detection.

Proposed method

• Fine-tuned RoBERTa architecture on a large-scale Dutch text corpus exceeding 32 GB of web-crawled data.
• Utilized masked language modeling and next sentence prediction pretraining objectives to learn contextual representations.
• Fine-tuned the pre-trained belabBERT model on psychiatric classification datasets for downstream task performance.
• Compared classification performance of belabBERT against RobBERT on text-only tasks using standard evaluation metrics.
• Evaluated audio-based classification models using speech features for the same psychiatric classification task.
• Explored a hybrid framework combining text embeddings from belabBERT and audio embeddings for joint classification.

Experimental results

Research questions

• RQ1 Can a large-scale, RoBERTa-based language model like belabBERT improve psychiatric classification performance in Dutch compared to existing models?
• RQ2 How does text-based classification using belabBERT compare to audio-based classification for detecting psychiatric disorders?
• RQ3 To what extent can combining text and audio features enhance classification performance in psychiatric diagnosis?
• RQ4 Does the use of a larger, more diverse Dutch pretraining corpus improve long-range semantic understanding in psychiatric text?

Key findings

• belabBERT outperformed the existing state-of-the-art Dutch model, RobBERT, in text-based psychiatric classification tasks.
• The performance of belabBERT surpassed that of audio-only classification models, indicating the strength of textual features for this task.
• The hybrid text- and audio-based classification framework showed promise, though specific quantitative improvements were not detailed.
• The model demonstrated enhanced capture of long-range semantic dependencies due to its large-scale pretraining on diverse Dutch text.
• The results confirm the value of large-scale, multilingual pretraining for low-resource languages like Dutch in clinical NLP applications.
• Substantial text overlap with a prior work (arXiv:2008.01543) was noted, suggesting incremental development of the model architecture.