[Paper Review] WLV-RIT at GermEval 2021: Multitask Learning with Transformers to Detect Toxic, Engaging, and Fact-Claiming Comments

Skye Morgan, Tharindu Ranasinghe+1 more|arXiv (Cornell University)|Jul 30, 2021

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper presents WLV-RIT's submission to GermEval 2021, proposing a multitask learning (MTL) approach using German-optimized transformer models—gBERT and gELECTRA—to jointly detect toxic, engaging, and fact-claiming comments. MTL significantly outperformed single-task learning across all three tasks, with the best system achieving an F1 score of 0.7653 on fact-claiming and 0.7198 on engaging comments.

ABSTRACT

This paper addresses the identification of toxic, engaging, and fact-claiming comments on social media. We used the dataset made available by the organizers of the GermEval-2021 shared task containing over 3,000 manually annotated Facebook comments in German. Considering the relatedness of the three tasks, we approached the problem using large pre-trained transformer models and multitask learning. Our results indicate that multitask learning achieves performance superior to the more common single task learning approach in all three tasks. We submit our best systems to GermEval-2021 under the team name WLV-RIT.

Motivation & Objective

• To address the challenge of detecting multiple types of user comments—toxic, engaging, and fact-claiming—on German social media platforms.
• To investigate whether multitask learning (MTL) with shared transformer representations improves performance over single-task learning (STL) for these related classification tasks.
• To evaluate the impact of pre-trained language modeling (LM) fine-tuning on model performance in the context of MTL.
• To compare the effectiveness of multilingual BERT (mBERT) against German-specific large language models (gBERT and gELECTRA) on this shared task.
• To submit the best-performing systems to the GermEval 2021 shared task and analyze their performance on the test set.

Proposed method

• Fine-tuned large pre-trained transformer models—mBERT, gBERT, and gELECTRA—on the GermEval 2021 dataset for three related classification tasks.
• Implemented a multitask learning (MTL) framework where a single model predicts all three labels (toxic, engaging, fact-claiming) simultaneously.
• Compared MTL with single-task learning (STL) to assess performance gains from shared representation learning.
• Integrated masked language modeling (MLM) pre-training as an auxiliary objective (LM) during fine-tuning to improve generalization.
• Used majority voting from five models with different random seeds to enhance prediction robustness.
• Optimized hyperparameters (initial learning rate = 1e-5, number of epochs = 3, batch size = 8) via validation set performance and applied early stopping.

Experimental results

Research questions

• RQ1 Does multitask learning with shared transformer representations improve performance on detecting toxic, engaging, and fact-claiming comments compared to single-task learning?
• RQ2 How do German-specific large language models (gBERT, gELECTRA) compare to multilingual BERT (mBERT) in this multilabel classification task?
• RQ3 To what extent does pre-training with masked language modeling (LM) improve performance in the multitask setting?
• RQ4 Which combination of model architecture and training strategy (MTL vs. STL, LM vs. no LM) yields the highest F1 scores across all three tasks?
• RQ5 Can shared learning across related tasks reduce computational cost while improving performance in low-resource NLP scenarios like German social media text?

Key findings

• Multitask learning (MTL) consistently outperformed single-task learning (STL) across all three tasks, with the highest F1 scores achieved in MTL settings.
• The gELECTRA model with both language modeling (LM) and MTL achieved the best overall performance, scoring 0.7342 F1 on toxic comments, 0.7198 on engaging comments, and 0.7653 on fact-claiming comments.
• gELECTRA and gBERT outperformed mBERT across all tasks, indicating that language-specific pre-training improves performance on German text.
• The inclusion of language modeling (LM) during fine-tuning led to performance improvements in several configurations, particularly in the MTL setup.
• The best-performing models (gELECTRA with LM+MTL, gELECTRA with MTL, and gBERT with LM+MTL) ranked in the top three on the test set for all three tasks.
• The results confirm that shared representation learning via MTL enhances generalization and efficiency, especially when tasks are semantically related.