[Paper Review] Comparing BERT against traditional machine learning text classification

Santiago González-Carvajal, Eduardo C. Garrido‐Merchán|arXiv (Cornell University)|May 26, 2020

Topic ModelingComputer Science31 references186 citations

TL;DR

The paper empirically compares BERT with TF-IDF-based traditional ML methods across four language tasks, showing BERT generally achieves higher accuracy and easier implementation, highlighting transfer learning benefits.

ABSTRACT

The BERT model has arisen as a popular state-of-the-art machine learning model in the recent years that is able to cope with multiple NLP tasks such as supervised text classification without human supervision. Its flexibility to cope with any type of corpus delivering great results has make this approach very popular not only in academia but also in the industry. Although, there are lots of different approaches that have been used throughout the years with success. In this work, we first present BERT and include a little review on classical NLP approaches. Then, we empirically test with a suite of experiments dealing different scenarios the behaviour of BERT against the traditional TF-IDF vocabulary fed to machine learning algorithms. Our purpose of this work is to add empirical evidence to support or refuse the use of BERT as a default on NLP tasks. Experiments show the superiority of BERT and its independence of features of the NLP problem such as the language of the text adding empirical evidence to use BERT as a default technique to be used in NLP problems.

Motivation & Objective

• Assess whether BERT provides empirical advantages over TF-IDF-based traditional NLP approaches for text classification.
• Evaluate BERT's language independence across multiple languages.
• Demonstrate the practicality and ease of implementing BERT compared to traditional pipelines.
• Provide insights on transfer learning effects in low-resource or multilingual settings.

Proposed method

• Compare a pre-trained BERT model using ktrain against a TF-IDF vocabulary fed into various traditional ML classifiers.
• Use four datasets with different languages and classification tasks.
• Employ TF-IDF with standard ML models as baseline competitors.
• Utilize AutoML tools (Predictor, H2OAutoML) in some experiments to select strong traditional models.
• Describe BERT architecture and pre-training/fine-tuning workflow for context.

Experimental results

Research questions

• RQ1 Does BERT outperform TF-IDF-based traditional ML methods across multiple languages and domains?
• RQ2 Is BERT's performance robust to language and dataset size variations?
• RQ3 How does transfer learning influence performance in low-resource or multilingual text classification tasks?
• RQ4 What is the relative ease of implementing BERT versus building traditional NLP pipelines?

Key findings

• In IMDB sentiment classification, BERT achieves 0.9387 accuracy, outperforming Voting Classifier (0.9007) and other baselines such as Logistic Regression (0.8949) and Linear SVC (0.8989).
• In RealOrNot tweets, BERT attains 0.8361 accuracy (Kaggle score 0.83640) while the best H2OAutoML model scores 0.7875 accuracy (Kaggle 0.77607).
• In Portuguese news, BERT reaches 0.9093 accuracy (Kaggle 0.91196) versus GradientBoostingClassifier at 0.85047 Kaggle.
• In Chinese hotel reviews, BERT achieves 0.9381 accuracy, while Predictor (auto_ml) scores 0.7399; the traditional best model is GradientBoostingClassifier.
• Across the four experiments, BERT generally outperforms traditional TF-IDF-based approaches, and implementing BERT is reported as less complicated.