[Paper Review] Identifying and Compensating for Feature Deviation in Imbalanced Deep Learning

Han-Jia Ye, Hong-You Chen+2 more|arXiv (Cornell University)|Jan 6, 2020

Imbalanced Data Classification Techniques62 references55 citations

TL;DR

The paper identifies a feature deviation phenomenon causing over-fitting to minor classes in imbalanced deep learning and proposes class-dependent temperatures (CDT) to compensate during training, improving minor-class performance on benchmarks.

ABSTRACT

Classifiers trained with class-imbalanced data are known to perform poorly on test data of the "minor" classes, of which we have insufficient training data. In this paper, we investigate learning a ConvNet classifier under such a scenario. We found that a ConvNet significantly over-fits the minor classes, which is quite opposite to traditional machine learning algorithms that often under-fit minor classes. We conducted a series of analysis and discovered the feature deviation phenomenon -- the learned ConvNet generates deviated features between the training and test data of minor classes -- which explains how over-fitting happens. To compensate for the effect of feature deviation which pushes test data toward low decision value regions, we propose to incorporate class-dependent temperatures (CDT) in training a ConvNet. CDT simulates feature deviation in the training phase, forcing the ConvNet to enlarge the decision values for minor-class data so that it can overcome real feature deviation in the test phase. We validate our approach on benchmark datasets and achieve promising performance. We hope that our insights can inspire new ways of thinking in resolving class-imbalanced deep learning.

Motivation & Objective

• Understand why ConvNets over-fit minor classes under long-tailed distributions.
• Characterize the feature deviation between training and test data for minor classes.
• Evaluate limitations of re-weighting and re-sampling in addressing imbalanced learning for ConvNets.
• Propose and validate a training strategy (CDT) that compensates for feature deviation without reducing it.

Proposed method

• Empirically analyze ConvNet behavior on imbalanced data and observe feature deviation between training and test features for minor classes.
• Decompose the classifier as ŷ = arg max_c w_c^T f_theta(x) and study how minor classes deviate in feature space.
• Quantify feature deviation via distances between class-wise training and test feature means (Equation 4).
• Introduce class-dependent temperatures a_c to modify the training objective, effectively enlarging minor-class decision values.
• Define a_c = (N_max / N_c)^gamma with gamma >= 0 to control the degree of compensation, and train with modified cross-entropy (Equation 5).
• Evaluate CDT across CIFAR-10/100, Tiny-ImageNet, and iNaturalist with varying imbalance ratios, comparing to ERM, re-sampling, and re-weighting baselines.

Experimental results

Research questions

• RQ1 What causes poor minor-class performance in imbalanced deep learning with ConvNets?
• RQ2 Does feature deviation between training and test data explain over-fitting to minor classes?
• RQ3 Can adjusting the training objective to simulate feature deviation (CDT) improve test performance without reducing deviation itself?
• RQ4 How does CDT compare to re-sampling and re-weighting across standard imbalanced benchmarks?

Key findings

• ConvNets trained end-to-end on imbalanced data over-fit minor classes, unlike some traditional methods.
• Minor-class features diverge between training and test sets, with deviation increasing as class frequency decreases.
• Naive re-sampling and re-weighting do not reduce feature deviation and can fail to improve, or even worsen, minor-class performance.
• Class-dependent temperatures (CDT) compensate for feature deviation by enlarging minor-class decision values during training, improving test accuracy.
• CDT achieves superior or competitive performance compared to state-of-the-art methods on several benchmarks (CIFAR-10/100, Tiny-ImageNet, iNaturalist).