[Paper Review] Diverse mini-batch Active Learning

Fedor Zhdanov|arXiv (Cornell University)|Jan 17, 2019

Machine Learning and Algorithms15 references92 citations

TL;DR

The paper proposes a scalable mini-batch active learning method that combines informativeness with diversity by using weighted K-means clustering to select diverse, informative examples for labeling.

ABSTRACT

We study the problem of reducing the amount of labeled training data required to train supervised classification models. We approach it by leveraging Active Learning, through sequential selection of examples which benefit the model most. Selecting examples one by one is not practical for the amount of training examples required by the modern Deep Learning models. We consider the mini-batch Active Learning setting, where several examples are selected at once. We present an approach which takes into account both informativeness of the examples for the model, as well as the diversity of the examples in a mini-batch. By using the well studied K-means clustering algorithm, this approach scales better than the previously proposed approaches, and achieves comparable or better performance.

Motivation & Objective

• Reduce labeled data needed to train supervised models.
• Address practical constraints of retraining with deep models by selecting mini-batches.
• Incorporate both informativeness and diversity in batch selection.
• Provide a scalable solution leveraging K-means clustering.
• Demonstrate effectiveness across text and image datasets with various models.

Proposed method

• Formulate batch selection as a facility location problem to improve diversity.
• Approximate with K-means clustering to achieve scalability over submodular methods.
• Incorporate informativeness scores by weighting cluster centers in a weighted K-means objective.
• Use margin-based uncertainty as the informativeness measure.
• Pre-filter unlabeled examples to a subset before clustering for efficiency.
• Select k examples closest to cluster centers for labeling in each batch.

Experimental results

Research questions

• RQ1 Does integrating diversity with informativeness in mini-batch selection improve learning efficiency compared to uncertainty-only baselines?
• RQ2 Can K-means clustering provide a scalable approximation to diversity-aware batch selection for active learning?
• RQ3 How does the proposed approach perform across text and image datasets with different model architectures?
• RQ4 What is the impact of pre-filtering parameter beta on performance and scalability?
• RQ5 Is margin-based uncertainty more effective than entropy-based or other uncertainty measures in this setting?

Key findings

• Diversity-aware mini-batch selection generally outperforms uncertainty sampling across multiple datasets.
• Clustering-based methods are significantly faster than submodular optimization approaches while achieving comparable or better accuracy.
• Weighted clustering leveraging informativeness scores improves performance on several datasets.
• First batch selection via clustering can boost early accuracy for some datasets.
• CIFAR-10 results show diversity-based methods slightly outperform plain uncertainty, with weighted clustering often best.
• Overall, the approach is scalable and competitive with more complex techniques while remaining simpler to implement.