[Paper Review] BERT for Sentiment Analysis: Pre-trained and Fine-Tuned Alternatives

Frederico Souza, João Filho|arXiv (Cornell University)|Jan 10, 2022

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper evaluates multiple strategies for aggregating BERT embeddings—particularly BERTimbau and m-BERT—for sentiment analysis in Brazilian Portuguese, demonstrating that combining CLS token with mean and standard deviation of all tokens significantly improves performance over using CLS alone. Fine-tuning further boosts results, with BERTimbau outperforming m-BERT across all datasets, achieving state-of-the-art ROC-AUC scores.

ABSTRACT

BERT has revolutionized the NLP field by enabling transfer learning with large language models that can capture complex textual patterns, reaching the state-of-the-art for an expressive number of NLP applications. For text classification tasks, BERT has already been extensively explored. However, aspects like how to better cope with the different embeddings provided by the BERT output layer and the usage of language-specific instead of multilingual models are not well studied in the literature, especially for the Brazilian Portuguese language. The purpose of this article is to conduct an extensive experimental study regarding different strategies for aggregating the features produced in the BERT output layer, with a focus on the sentiment analysis task. The experiments include BERT models trained with Brazilian Portuguese corpora and the multilingual version, contemplating multiple aggregation strategies and open-source datasets with predefined training, validation, and test partitions to facilitate the reproducibility of the results. BERT achieved the highest ROC-AUC values for the majority of cases as compared to TF-IDF. Nonetheless, TF-IDF represents a good trade-off between the predictive performance and computational cost.

Motivation & Objective

• To investigate alternative aggregation strategies for BERT output embeddings in sentiment analysis, moving beyond the standard [CLS] token approach.
• To compare the performance of language-specific BERT models (BERTimbau) versus multilingual models (m-BERT) on Brazilian Portuguese text classification.
• To evaluate the impact of fine-tuning versus pre-trained inference on sentiment classification accuracy.
• To assess cross-dataset generalization performance of BERT models trained on one dataset and tested on others.
• To provide practical guidelines for deploying BERT in low-resource or computationally constrained environments using 16-bit precision and efficient aggregation.

Proposed method

• Employed BERTimbau Base and Large, and multilingual BERT (m-BERT) models fine-tuned on Brazilian Portuguese corpora.
• Proposed and evaluated multiple aggregation strategies: [CLS] token only, mean of all tokens, standard deviation, and combinations like 'first + mean + std'.
• Used Logistic Regression as the classifier on aggregated embeddings for all experiments.
• Conducted experiments with both pre-trained and fine-tuned BERT models using different learning and dropout rates.
• Evaluated models on five open-source datasets with predefined train/val/test splits to ensure reproducibility.
• Applied 16-bit floating-point precision (float16) to reduce memory and computation costs without significant performance loss.

Experimental results

Research questions

• RQ1 Does combining multiple BERT output embeddings (e.g., CLS, mean, std) yield better sentiment classification performance than using only the [CLS] token?
• RQ2 How do language-specific BERT models (BERTimbau) compare to multilingual models (m-BERT) in sentiment analysis for Brazilian Portuguese?
• RQ3 To what extent does fine-tuning BERT improve performance on sentiment classification compared to using pre-trained embeddings?
• RQ4 How well do models fine-tuned on one dataset generalize to other datasets in cross-dataset evaluation?
• RQ5 What is the trade-off between computational cost and performance when using 16-bit precision versus 32-bit in BERT inference?

Key findings

• The aggregation strategy 'first + mean + std' significantly outperformed the standard [CLS] token approach across all datasets, improving ROC-AUC by up to 3.4 percentage points.
• BERTimbau Base and Large models achieved the highest ROC-AUC scores on all datasets, outperforming m-BERT, with BERTimbau Large reaching 99.2% AUC on the B2W dataset.
• Fine-tuning led to substantial performance gains: UTLC-Movies saw a 10% ROC-AUC improvement with m-BERT and 3.4% with BERTimbau Large compared to pre-trained models.
• The generalist model trained on all datasets combined performed better than the pre-trained model but worse than specialized fine-tuned models, indicating that domain-specific fine-tuning remains valuable.
• Using 16-bit precision reduced memory and training time significantly with no measurable drop in performance, making it a practical choice for resource-constrained deployments.
• Cross-dataset fine-tuning showed modest gains (e.g., 1.3 percentage points on UTLC-Movies when fine-tuned on B2W), but overall gains were less than 1% for most cases, suggesting limited cost-effectiveness for such transfers.