[Paper Review] Exploring Transformers in Emotion Recognition: a comparison of BERT, DistillBERT, RoBERTa, XLNet and ELECTRA

Diogo Cortiz|arXiv (Cornell University)|Apr 5, 2021

Emotion and Mood Recognition20 citations

TL;DR

This study evaluates five state-of-the-art transformer models—BERT, DistilBERT, RoBERTa, XLNet, and ELECTRA—for fine-grained emotion recognition using a dedicated dataset. It compares their performance via F1-score and inference speed, finding that RoBERTa and ELECTRA achieve the highest accuracy, while DistilBERT offers a strong speed-accuracy trade-off.

ABSTRACT

This paper investigates how Natural Language Understanding (NLU) could be applied in Emotion Recognition, a specific task in affective computing. We finetuned different transformers language models (BERT, DistilBERT, RoBERTa, XLNet, and ELECTRA) using a fine-grained emotion dataset and evaluating them in terms of performance (f1-score) and time to complete.

Motivation & Objective

• To assess the effectiveness of pre-trained transformer models in fine-grained emotion recognition tasks.
• To compare the performance and inference efficiency of BERT, DistilBERT, RoBERTa, XLNet, and ELECTRA on a shared emotion dataset.
• To identify the optimal balance between model accuracy and computational efficiency for emotion recognition applications.
• To provide empirical insights into how architectural differences affect emotion classification outcomes.

Proposed method

• Fine-tuned five transformer models—BERT, DistilBERT, RoBERTa, XLNet, and ELECTRA—on a fine-grained emotion dataset using standard transfer learning protocols.
• Used standard hyperparameters and training procedures for each model, ensuring consistent evaluation conditions.
• Evaluated models using the F1-score as the primary metric to assess classification performance on emotion labels.
• Measured and compared inference time to assess computational efficiency and real-time applicability.
• Employed standard tokenization and sequence classification head for all models, maintaining consistency across architectures.
• Used a single dataset for all comparisons to ensure fair and direct model evaluation.

Experimental results

Research questions

• RQ1 Which transformer model achieves the highest F1-score in fine-grained emotion recognition?
• RQ2 How do the inference speeds of BERT, DistilBERT, RoBERTa, XLNet, and ELECTRA compare in emotion classification?
• RQ3 Does model compression (e.g., DistilBERT) lead to a significant drop in performance compared to full-sized models?
• RQ4 How do autoregressive (XLNet) and autoencoding (BERT, RoBERTa) architectures perform relative to each other in emotion recognition?
• RQ5 Do models with more parameters or more training data (e.g., RoBERTa, ELECTRA) consistently outperform smaller or less pre-trained models?

Key findings

• RoBERTa achieved the highest F1-score among all models, indicating superior performance in fine-grained emotion classification.
• ELECTRA also achieved a high F1-score, closely matching RoBERTa and outperforming BERT and XLNet.
• DistilBERT achieved a strong F1-score that was only slightly lower than RoBERTa and ELECTRA, demonstrating high efficiency.
• Despite its smaller size, DistilBERT showed significantly faster inference times than all other models, making it suitable for real-time applications.
• BERT and XLNet showed lower F1-scores compared to RoBERTa and ELECTRA, indicating reduced effectiveness for this specific task.
• The performance ranking across models was: RoBERTa ≈ ELECTRA > DistilBERT > BERT > XLNet in terms of F1-score.