[Paper Review] Low-Shot Classification: A Comparison of Classical and Deep Transfer Machine Learning Approaches

P.N.R. Usherwood, Steven Smit|arXiv (Cornell University)|Jul 17, 2019

Topic Modeling20 references4 citations

TL;DR

This paper compares deep transfer learning (e.g., BERT) and classical machine learning (e.g., SVM, logistic regression) in low-shot text classification with 100–1000 labeled examples per class. BERT outperforms classical methods by 9.7% on average at 100 examples, with significantly better cross-domain robustness, though classical models remain viable for resource-constrained settings.

ABSTRACT

Despite the recent success of deep transfer learning approaches in NLP, there is a lack of quantitative studies demonstrating the gains these models offer in low-shot text classification tasks over existing paradigms. Deep transfer learning approaches such as BERT and ULMFiT demonstrate that they can beat state-of-the-art results on larger datasets, however when one has only 100-1000 labelled examples per class, the choice of approach is less clear, with classical machine learning and deep transfer learning representing valid options. This paper compares the current best transfer learning approach with top classical machine learning approaches on a trinary sentiment classification task to assess the best paradigm. We find that BERT, representing the best of deep transfer learning, is the best performing approach, outperforming top classical machine learning algorithms by 9.7% on average when trained with 100 examples per class, narrowing to 1.8% at 1000 labels per class. We also show the robustness of deep transfer learning in moving across domains, where the maximum loss in accuracy is only 0.7% in similar domain tasks and 3.2% cross domain, compared to classical machine learning which loses up to 20.6%.

Motivation & Objective

• To evaluate whether deep transfer learning offers measurable advantages over classical machine learning in low-shot text classification settings with limited labeled data (100–1000 examples per class).
• To assess the robustness of both paradigms across intra- and inter-domain transfer learning scenarios.
• To provide empirical guidance for practitioners choosing between classical and deep transfer learning in real-world NLP applications with small datasets.
• To investigate the trade-offs between model performance, computational cost, and deployment feasibility in low-shot settings.

Proposed method

• Trained state-of-the-art deep transfer learning models (BERT and ULMFiT) on a trinary sentiment classification task using fine-tuning on small labeled datasets.
• Compared with top classical machine learning models (SVM, logistic regression, fastText) using standard NLP preprocessing and feature engineering.
• Evaluated performance across multiple public datasets to ensure generalization beyond a single domain or data distribution.
• Conducted hyperparameter tuning for each model class to ensure optimal configuration before comparison.
• Measured performance using accuracy on both in-domain and cross-domain test sets to assess transferability.
• Used BERT’s multilingual and domain-adapted variants to evaluate robustness to vocabulary and domain shifts.

Experimental results

Research questions

• RQ1 How do deep transfer learning models like BERT compare to classical machine learning models in terms of accuracy on low-shot text classification tasks with 100–1000 labeled examples per class?
• RQ2 How does the performance of each paradigm vary when transferring models across domains with differing text characteristics (e.g., length, vocabulary, slang)?
• RQ3 What is the relative robustness of classical and deep transfer learning models when tested on out-of-domain data?
• RQ4 What are the practical trade-offs between model performance, inference cost, and human effort (e.g., hyperparameter tuning, feature engineering) in low-shot settings?

Key findings

• BERT outperformed top classical machine learning models by 9.7% on average when trained with only 100 labeled examples per class.
• The performance gap narrowed to 1.8% when training data increased to 1000 examples per class, indicating diminishing returns of transfer learning at scale.
• Deep transfer learning models showed strong cross-domain robustness, with a maximum accuracy drop of only 3.2% across domains, compared to up to 20.6% for classical models.
• Classical models were more sensitive to domain shifts, particularly in out-of-vocabulary token handling, especially in domains with slang or informal language.
• BERT’s WordPiece tokenization provided better handling of rare or out-of-vocabulary words than standard subword or word-level tokenization in classical models.
• Despite higher hardware requirements (e.g., 12GB VRAM for BERT-base), deep transfer learning reduced human effort in feature engineering and hyperparameter tuning, offering long-term cost savings in production.