[Paper Review] A Comparative Survey of Deep Active Learning

Xueying Zhan, Qingzhong Wang+4 more|arXiv (Cornell University)|Mar 25, 2022

Machine Learning and Algorithms48 citations

TL;DR

The paper builds the DeepAL+ toolkit by re-implementing 19 DAL methods, conducts comprehensive cross-dataset comparisons, and analyzes factors like batch size and training epochs to guide fair, effective DAL experiments.

ABSTRACT

While deep learning (DL) is data-hungry and usually relies on extensive labeled data to deliver good performance, Active Learning (AL) reduces labeling costs by selecting a small proportion of samples from unlabeled data for labeling and training. Therefore, Deep Active Learning (DAL) has risen as a feasible solution for maximizing model performance under a limited labeling cost/budget in recent years. Although abundant methods of DAL have been developed and various literature reviews conducted, the performance evaluation of DAL methods under fair comparison settings is not yet available. Our work intends to fill this gap. In this work, We construct a DAL toolkit, DeepAL+, by re-implementing 19 highly-cited DAL methods. We survey and categorize DAL-related works and construct comparative experiments across frequently used datasets and DAL algorithms. Additionally, we explore some factors (e.g., batch size, number of epochs in the training process) that influence the efficacy of DAL, which provides better references for researchers to design their DAL experiments or carry out DAL-related applications.

Motivation & Objective

• Motivate fair and comprehensive evaluation of Deep Active Learning (DAL) methods under labeling budgets.
• Provide a reproducible DAL benchmarking platform by re-implementing leading DAL methods in DeepAL+.
• Survey DAL methods by querying strategy (uncertainty, representativeness/diversity, combined) and enhancement techniques.
• Assess the impact of training settings (batch size, epochs) and data/model enhancements on DAL performance across tasks.
• Offer guidelines for designing fair DAL experiments and identify challenges and future research directions.

Proposed method

• Re-implementation of 19 highly-cited DAL methods within the DeepAL+ toolkit.
• Pool-based active learning formulation with a labeled set LC and unlabeled pool LU.
• Classification-focused tasks with conventional backbones (e.g., ResNet18) to ensure fair comparisons.
• Categorization of querying strategies into uncertainty-based, representative/diversity-based, and combined approaches.
• Evaluation across multiple datasets (e.g., MNIST, FashionMNIST, EMNIST, SVHN, CIFAR variants, TinyImageNet, BreakHis, Pneumonia-MNIST, Waterbird).
• Ablation studies to analyze effects of training epochs and batch size; timing and memory considerations; and pre-training impact.

Experimental results

Research questions

• RQ1 How do different DAL querying strategies (uncertainty-based, representativeness-based, and combined) compare under equal training and budgeting conditions?
• RQ2 What is the impact of training hyper-parameters (number of epochs, batch size) on DAL performance across tasks?
• RQ3 Do data/model enhancement techniques (pseudo-labeling, data augmentation, ensembles, loss-function tweaks) consistently improve DAL performance across datasets?
• RQ4 How does pre-training influence the effectiveness of DAL methods on tasks with spurious correlations or varied semantics (e.g., Waterbird, MNIST)?

Key findings

• Uncertainty-based DAL methods offer modest improvements (~1-3% on average) over Random on standard image classification tasks, but no single method dominates across all datasets.
• Two-stage and gradient-based methods (e.g., BADGE, WAAL) show competitive performance with favorable trade-offs in some tasks, particularly CIFAR100.
• Representativeness/diversity methods (KMeans, VAAL, VAAL variants) often incur higher memory/time costs and provide less consistent gains than uncertainty-based methods on several datasets.
• Pre-training can improve DAL performance in some cases (e.g., Waterbird) by providing better feature representations, though effects vary by dataset and task.
• Ablation studies indicate more training epochs can boost performance for some methods, but gains exhibit diminishing returns; batch size has variable impact with larger batches sometimes reducing iteration frequency benefits.
• Enhanced DAL approaches (LPL, WAAL, AdvBIM, CEAL) show potential but can be sensitive to hyperparameters and dataset characteristics, with mixed results across tasks.