[Paper Review] Deep Active Learning with Augmentation-based Consistency Estimation

SeulGi Hong, Heonjin Ha+2 more|arXiv (Cornell University)|Nov 5, 2020

Machine Learning and Algorithms22 references4 citations

TL;DR

This paper proposes a deep active learning framework that integrates data augmentation-based consistency estimation to improve model generalization. By redefining Cutout and CutMix as quantitative regularization losses and uncertainty metrics, the method enhances performance across diverse datasets and labeling budgets, achieving consistent gains when combined with existing uncertainty sampling strategies.

ABSTRACT

In active learning, the focus is mainly on the selection strategy of unlabeled data for enhancing the generalization capability of the next learning cycle. For this, various uncertainty measurement methods have been proposed. On the other hand, with the advent of data augmentation metrics as the regularizer on general deep learning, we notice that there can be a mutual influence between the method of unlabeled data selection and the data augmentation-based regularization techniques in active learning scenarios. Through various experiments, we confirmed that consistency-based regularization from analytical learning theory could affect the generalization capability of the classifier in combination with the existing uncertainty measurement method. By this fact, we propose a methodology to improve generalization ability, by applying data augmentation-based techniques to an active learning scenario. For the data augmentation-based regularization loss, we redefined cutout (co) and cutmix (cm) strategies as quantitative metrics and applied at both model training and unlabeled data selection steps. We have shown that the augmentation-based regularizer can lead to improved performance on the training step of active learning, while that same approach can be effectively combined with the uncertainty measurement metrics proposed so far. We used datasets such as FashionMNIST, CIFAR10, CIFAR100, and STL10 to verify the performance of the proposed active learning technique for multiple image classification tasks. Our experiments show consistent performance gains for each dataset and budget scenario.

Motivation & Objective

• To address the limited generalization of active learning methods across different datasets and labeling budgets.
• To integrate data augmentation-based regularization techniques—specifically Cutout and CutMix—into active learning for improved model robustness.
• To redefine consistency-based regularization and uncertainty estimation using analytical learning theory to enhance performance in deep active learning.
• To demonstrate that augmentation-based regularization synergizes with existing uncertainty sampling methods, leading to consistent performance gains.
• To validate the method on multiple image classification benchmarks (FashionMNIST, CIFAR10, CIFAR100, STL10) under varying budget constraints.

Proposed method

• Reformulate Cutout and CutMix as differentiable, quantitative consistency-based regularization losses for use during model training in active learning.
• Introduce augmentation-based uncertainty estimation by measuring prediction variance across augmented views of unlabeled data to guide selection.
• Apply both regularization and uncertainty estimation at every active learning cycle—during model training and unlabeled data selection.
• Use analytical learning theory to justify that function variation under data augmentation correlates with generalization error, enabling consistency estimation.
• Combine the proposed regularization loss with standard uncertainty sampling methods (e.g., entropy, margin) to enhance performance.
• Train models using mini-batches with augmented data and optimize for both consistency (via regularization) and uncertainty-aware selection.

Experimental results

Research questions

• RQ1 Can data augmentation-based consistency estimation improve generalization in deep active learning?
• RQ2 How does combining augmentation-based regularization with uncertainty sampling affect performance across different datasets?
• RQ3 Does the proposed method maintain performance gains under varying labeling budget constraints?
• RQ4 Can the proposed consistency-based regularization be effectively combined with existing uncertainty measurement techniques?
• RQ5 Is the performance improvement robust across diverse image classification datasets with varying class imbalance and data scarcity?

Key findings

• The proposed method achieves consistent performance gains across all evaluated datasets—FashionMNIST, CIFAR10, CIFAR100, and STL10—under various labeling budgets.
• Combining augmentation-based regularization with uncertainty sampling (e.g., entropy, margin) leads to significant and stable improvements over baseline methods.
• On CIFAR100 and STL10, which have fewer images per class, the combination of uncertainty and regularization yields higher performance than either component alone.
• The method shows robustness to budget changes, with performance improving steadily as more labeled data is acquired, especially when using the proposed regularization.
• The integration of Cutout and CutMix as consistency-based regularizers leads to better generalization than random sampling or isolated use of either technique.
• Experiments confirm that the synergy between uncertainty estimation and regularization is essential, as standalone use of either component yields limited gains.