[Paper Review] Is margin all you need? An extensive empirical study of active learning on tabular data

Dara Bahri, Heinrich Jiang+2 more|arXiv (Cornell University)|Oct 7, 2022

Machine Learning and Algorithms4 citations

TL;DR

This paper evaluates 15 active learning methods on 69 real-world tabular datasets using deep neural networks, finding that margin sampling—despite its simplicity—consistently matches or outperforms all state-of-the-art alternatives across diverse data regimes and pre-training settings. The key contribution is empirical evidence that margin sampling is a robust, hyperparameter-free baseline for tabular active learning.

ABSTRACT

Given a labeled training set and a collection of unlabeled data, the goal of active learning (AL) is to identify the best unlabeled points to label. In this comprehensive study, we analyze the performance of a variety of AL algorithms on deep neural networks trained on 69 real-world tabular classification datasets from the OpenML-CC18 benchmark. We consider different data regimes and the effect of self-supervised model pre-training. Surprisingly, we find that the classical margin sampling technique matches or outperforms all others, including current state-of-art, in a wide range of experimental settings. To researchers, we hope to encourage rigorous benchmarking against margin, and to practitioners facing tabular data labeling constraints that hyper-parameter-free margin may often be all they need.

Motivation & Objective

• To rigorously evaluate the performance of multiple active learning methods on real-world tabular datasets under diverse training regimes.
• To assess whether modern, complex active learning methods consistently outperform the classical margin sampling technique in practical settings.
• To investigate the impact of model pre-training, seed set size, and batch size on active learning performance.
• To provide a comprehensive benchmark for future research and practical deployment in tabular data active learning.
• To challenge the assumption that more complex active learning methods are inherently superior to simple uncertainty-based strategies.

Proposed method

• The study evaluates 15 active learning methods, including uncertainty-based, diversity-based, and uncertainty+diversity hybrids, on 69 tabular datasets from the OpenML-CC18 benchmark.
• Models are trained using deep neural networks with stochastic gradient descent, and active learning is performed in iterative rounds with fixed batch sizes.
• Margin sampling selects instances with the smallest difference between the top two softmax probabilities, representing highest prediction uncertainty.
• Performance is measured using test accuracy at multiple acquisition rounds, with statistical significance assessed via paired t-tests at p ≤ 0.1.
• The analysis includes both pre-trained and non-pretrained models, with results reported across small, medium, and large data regimes.
• Win matrices and box plots are used to compare relative performance, with relative improvement over random sampling visualized across datasets.

Experimental results

Research questions

• RQ1 Does margin sampling outperform modern, state-of-the-art active learning methods on real-world tabular datasets when using deep neural networks?
• RQ2 How do different data regimes (small, medium, large seed sets and batch sizes) affect the relative performance of active learning methods?
• RQ3 Does self-supervised pre-training improve the performance of non-margin active learning methods relative to margin?
• RQ4 Are diversity-promoting or uncertainty+diversity methods consistently superior to margin sampling in practical settings?
• RQ5 Is the performance advantage of complex methods statistically significant across a broad range of datasets and configurations?

Key findings

• Margin sampling outperforms all other active learning methods in terms of test accuracy across all data regimes and pre-training conditions.
• No method significantly outperformed margin in a statistically meaningful way, even when using self-supervised pre-training.
• The relative improvement of margin over random sampling was 1–3% on average across datasets in the medium setting, with consistent gains across acquisition rounds.
• Methods like BALD and PowerBALD underperformed margin, with PowerBALD’s performance likely due to its noise mechanism mimicking random sampling.
• Clustering-based methods like Cluster-Margin and TypiClust performed similarly to random or worse, except when cluster size was very small (1.25), where performance approached margin.
• Even in the small setup with only 30 initial labeled examples, margin still provided a consistent performance boost over random sampling, indicating robustness to low data regimes.