[Paper Review] Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples

Haw-Shiuan Chang, Erik Learned-Miller+1 more|arXiv (Cornell University)|Apr 24, 2017

Machine Learning and Algorithms46 references130 citations

TL;DR

The paper introduces two lightweight active-bias reweighting strategies that emphasize uncertain samples during SGD to improve neural network accuracy and robustness across datasets and architectures.

ABSTRACT

Self-paced learning and hard example mining re-weight training instances to improve learning accuracy. This paper presents two improved alternatives based on lightweight estimates of sample uncertainty in stochastic gradient descent (SGD): the variance in predicted probability of the correct class across iterations of mini-batch SGD, and the proximity of the correct class probability to the decision threshold. Extensive experimental results on six datasets show that our methods reliably improve accuracy in various network architectures, including additional gains on top of other popular training techniques, such as residual learning, momentum, ADAM, batch normalization, dropout, and distillation.

Motivation & Objective

• Motivate robust learning by balancing easy and hard examples without relying on label noise assumptions.
• Develop two lightweight uncertainty-based methods to bias SGD: prediction-variance and threshold-closeness based reweighting.
• Demonstrate that active bias improves generalization across multiple datasets and architectures.
• Show compatibility with existing training techniques and effectiveness under different noise conditions.

Proposed method

• Propose two active bias methods that emphasize uncertain samples during mini-batch SGD: SGD-WPV (Weighted by Prediction Variance) and SGD-WTC (Weighted by Threshold Closeness).
• Estimate prediction variance from history of predicted probabilities to guide sampling or weighting; assume posterior parameter uncertainty after burn-in epochs.
• Use a threshold-closeness objective that weights samples by the product p(y|x)^*(1-p(y|x)) to target boundary samples.
• Provide a burn-in period to stabilize uncertainty estimates before applying bias.
• Compare sampling vs loss-weighting variants (SGD-SPV, SGD-WPV, SGD-STC, SGD-WTC) against uniform SGD, SGD-SD, and SGD-ISD baselines.
• Demonstrate compatibility with standard optimizers and training techniques (momentum, Adam, batch norm, dropout, distillation).

Experimental results

Research questions

• RQ1 Does emphasizing uncertain/high-variance samples during SGD improve generalization across tasks and architectures?
• RQ2 Are variance-based and threshold-based uncertainty measures effective and robust to noise and label corruption?
• RQ3 How do active-bias methods interact with common training techniques (e.g., momentum, ADAM, batch normalization, distillation)?
• RQ4 Is burn-in necessary for reliable uncertainty estimates when applying active bias?
• RQ5 Do the methods provide benefits when training on both easy and difficult/noisy datasets?

Key findings

• Active bias methods consistently outperform SGD-Uni and SGD-Scan across six datasets and multiple architectures.
• SGD-WPV and SGD-WD show robust improvements, including in noisy settings (e.g., Noisy MNIST).
• Threshold-based weighting (SGD-WTC/SGD-STC) achieves competitive or superior results in several experiments, including CIFAR and NLP tasks.
• With distillation, active bias methods (especially SGD-WTC) can match or exceed distillation alone, indicating compatibility.
• Improvements persist across simple logistic regression baselines to deep residual networks, suggesting broad applicability.