[Paper Review] GALAXY: Graph-based Active Learning at the Extreme

Jifan Zhang, Julian Katz-Samuels+1 more|arXiv (Cornell University)|Feb 3, 2022

Machine Learning and Algorithms4 citations

TL;DR

GALAXY is a graph-based active learning method that addresses extreme class imbalance by adaptively selecting uncertain yet class-balanced examples through a bisection-based sorting of uncertainty scores and label diversity. It outperforms state-of-the-art methods by reducing label queries by up to 30% while achieving significantly higher balanced accuracy in imbalanced vision benchmarks.

ABSTRACT

Active learning is a label-efficient approach to train highly effective models while interactively selecting only small subsets of unlabelled data for labelling and training. In "open world" settings, the classes of interest can make up a small fraction of the overall dataset -- most of the data may be viewed as an out-of-distribution or irrelevant class. This leads to extreme class-imbalance, and our theory and methods focus on this core issue. We propose a new strategy for active learning called GALAXY (Graph-based Active Learning At the eXtrEme), which blends ideas from graph-based active learning and deep learning. GALAXY automatically and adaptively selects more class-balanced examples for labeling than most other methods for active learning. Our theory shows that GALAXY performs a refined form of uncertainty sampling that gathers a much more class-balanced dataset than vanilla uncertainty sampling. Experimentally, we demonstrate GALAXY's superiority over existing state-of-art deep active learning algorithms in unbalanced vision classification settings generated from popular datasets.

Motivation & Objective

• Address the challenge of extreme class imbalance in active learning, where minority classes are rare and standard methods favor the majority class.
• Develop a scalable, efficient active learning algorithm that maintains class balance during label selection without prior knowledge of class distribution.
• Ensure that selected examples are both uncertain and representative of diverse classes, even when the overall dataset is heavily skewed.
• Provide theoretical justification that GALAXY achieves better class balance than standard uncertainty sampling.

Proposed method

• GALAXY sorts unlabeled examples by their softmax uncertainty scores and applies a bisection procedure to identify consecutive pairs with differing labels, promoting label diversity.
• It constructs a graph from deep features and uses the model’s predictions to guide the selection of uncertain yet class-diverse examples.
• The algorithm dynamically rebuilds the graph after each model retraining, adapting to updated predictions and improving selection quality over time.
• It integrates uncertainty sampling with graph-based diversity by focusing on uncertain points near class boundaries across multiple classes.
• The method uses a batch selection strategy that balances uncertainty and label diversity, avoiding redundant queries from the same class.
• It requires minimal modification to standard uncertainty sampling pipelines, making it easy to implement and deploy.

Experimental results

Research questions

• RQ1 Can active learning methods effectively select informative examples when the dataset contains extreme class imbalance, such as a 1:99 ratio between minority and majority classes?
• RQ2 Does a graph-based approach improve the class balance of selected batches compared to standard uncertainty sampling in deep active learning?
• RQ3 How does GALAXY compare to existing hybrid uncertainty-diversity methods like BADGE, BAIT, and SIMILAR in terms of label efficiency and accuracy under imbalance?
• RQ4 To what extent does the dynamic graph construction in GALAXY enhance selection quality compared to static graph methods like S²?
• RQ5 Can GALAXY maintain high performance across diverse datasets and varying batch sizes while reducing the number of required labels?

Key findings

• GALAXY reduces the number of label queries by up to 30% compared to the second-best method, achieving the same balanced accuracy in unbalanced SVHN (2 classes) and CIFAR-100 (3 classes).
• In unbalanced SVHN with 2 classes, GALAXY reaches 92% balanced accuracy with 1,700 queries, while the next best method requires 2,500 queries.
• In unbalanced CIFAR-100 with 3 classes, GALAXY achieves 66% balanced accuracy with 1,600 queries, outperforming the second-best method that requires 2,200 queries.
• GALAXY collects significantly more in-distribution labels than baseline methods, including uncertainty sampling and Most Likely Positive, which prioritize certainty over uncertainty.
• The method outperforms S² with both 1-nearest-neighbor and neural network models, demonstrating the importance of dynamic graph construction based on updated model predictions.
• Experiments show that GALAXY maintains superior performance across all 8 tested settings, including a large-budget regime on CIFAR-100 with 10 classes and a batch size of 1,000.