[Paper Review] Making the most of small Software Engineering datasets with modern machine learning

Julian Aron Prenner, Romain Robbes|arXiv (Cornell University)|Jun 29, 2021

Software Engineering Research4 citations

TL;DR

This paper evaluates pre-trained Transformer models—such as BERT and RoBERTa—on 13 small Software Engineering datasets (under 1,000 samples), demonstrating their effectiveness for natural language tasks and competitive performance on code-related tasks. It finds that while pre-training boosts performance on small SE datasets, traditional ML methods still outperform Transformers on very small code datasets, and recommends techniques like active learning and soft labels for improved learning efficiency.

ABSTRACT

This paper provides a starting point for Software Engineering (SE) researchers and practitioners faced with the problem of training machine learning models on small datasets. Due to the high costs associated with labeling data, in Software Engineering,there exist many small (< 1 000 samples) and medium-sized (< 100 000 samples) datasets. While deep learning has set the state of the art in many machine learning tasks, it is only recently that it has proven effective on small-sized datasets, primarily thanks to pre-training, a semi-supervised learning technique that leverages abundant unlabelled data alongside scarce labelled data.In this work, we evaluate pre-trained Transformer models on a selection of 13 smaller datasets from the SE literature, covering both,source code and natural language. Our results suggest that pre-trained Transformers are competitive and in some cases superior to previous models, especially for tasks involving natural language; whereas for source code tasks, in particular for very small datasets,traditional machine learning methods often has the edge.In addition, we experiment with several techniques that ought to aid training on small datasets, including active learning, data augmentation, soft labels, self-training and intermediate-task fine-tuning, and issue recommendations on when they are effective. We also release all the data, scripts, and most importantly pre-trained models for the community to reuse on their own datasets.

Motivation & Objective

• To address the challenge of training effective machine learning models on small Software Engineering datasets, which are common due to high labeling costs.
• To evaluate the effectiveness of pre-trained Transformer models (e.g., BERT, RoBERTa) on small SE datasets across both source code and natural language tasks.
• To investigate and recommend techniques—such as active learning, data augmentation, and self-training—that improve model performance on small datasets.
• To release all data, code, and pre-trained models to support reuse and reproducibility in the SE community.

Proposed method

• Fine-tuning pre-trained Transformer models (e.g., BERT, RoBERTa) on 13 small SE datasets from the literature, covering both code and natural language tasks.
• Applying data augmentation via back-translation and synonym replacement to increase training data diversity.
• Using soft labels derived from ensemble models to improve generalization on limited labeled data.
• Implementing active learning by iteratively selecting the most uncertain samples for labeling to reduce annotation cost.
• Performing intermediate-task fine-tuning on related NLP tasks before fine-tuning on target SE tasks to improve inductive bias.
• Evaluating model performance using standard metrics (e.g., F1, accuracy) and comparing against prior state-of-the-art models in each dataset.

Experimental results

Research questions

• RQ1 How do pre-trained Transformer models perform on small Software Engineering datasets compared to traditional machine learning models?
• RQ2 Which data augmentation and labeling techniques (e.g., back-translation, soft labels) most effectively improve model performance on small SE datasets?
• RQ3 In what scenarios does active learning reduce labeling effort while maintaining or improving model performance?
• RQ4 Does intermediate-task fine-tuning improve performance on small SE datasets, and for which tasks is it most beneficial?
• RQ5 Under what conditions do pre-trained models outperform or underperform compared to classical ML models in code-related SE tasks?

Key findings

• Pre-trained Transformers achieve competitive or superior performance on natural language tasks in small SE datasets, often outperforming prior state-of-the-art models.
• For very small code-related datasets (e.g., under 100 samples), traditional machine learning models such as SVMs and Random Forests still outperform Transformers.
• Active learning significantly reduces the number of required labeled examples while maintaining high performance, especially in low-resource settings.
• Soft labels derived from ensemble models improve generalization and model robustness on small datasets, particularly when labels are noisy or scarce.
• Data augmentation via back-translation improves performance on natural language tasks, though effects are more modest on code-related tasks.
• Intermediate-task fine-tuning improves performance on downstream SE tasks, especially when the intermediate task is semantically related to the target task.