[Paper Review] Balanced Meta-Softmax for Long-Tailed Visual Recognition

Jiawei Ren, Cunjun Yu+5 more|arXiv (Cornell University)|Jul 21, 2020

Domain Adaptation and Few-Shot Learning38 references317 citations

TL;DR

BALMS introduces Balanced Softmax and a Meta Sampler to address label distribution shift in long-tailed visual recognition, achieving state-of-the-art results on several datasets including LVIS.

ABSTRACT

Deep classifiers have achieved great success in visual recognition. However, real-world data is long-tailed by nature, leading to the mismatch between training and testing distributions. In this paper, we show that the Softmax function, though used in most classification tasks, gives a biased gradient estimation under the long-tailed setup. This paper presents Balanced Softmax, an elegant unbiased extension of Softmax, to accommodate the label distribution shift between training and testing. Theoretically, we derive the generalization bound for multiclass Softmax regression and show our loss minimizes the bound. In addition, we introduce Balanced Meta-Softmax, applying a complementary Meta Sampler to estimate the optimal class sample rate and further improve long-tailed learning. In our experiments, we demonstrate that Balanced Meta-Softmax outperforms state-of-the-art long-tailed classification solutions on both visual recognition and instance segmentation tasks.

Motivation & Objective

• Motivate and analyze why standard Softmax is biased under long-tailed data and how this harms generalization.
• Develop Balanced Softmax as an unbiased extension that accounts for train-test label distribution shift.
• Introduce a Meta Sampler to learn effective re-sampling and alleviate optimization issues in highly imbalanced data.
• Demonstrate empirical improvements across image classification and instance segmentation tasks on multiple long-tailed datasets.

Proposed method

• Derive Balanced Softmax by modeling the test-time balanced distribution and the imbalanced training distribution and formulating a corresponding cross-entropy loss.
• Show that optimizing Balanced Softmax cross-entropy minimizes a generalization error bound for multiclass Softmax regression.
• Propose Balanced Softmax as  lhat() = -log(n_y e^{ta_y} / sum_i n_i e^{ta_i}) to incorporate class frequencies n_j.
• Introduce Meta Sampler to learn class-wise sampling rates via a bi-level meta-learning objective with a meta-set supervised inner loop optimization.
• Utilize Gumbel-Softmax reparameterization to enable differentiable sampling during training.
• Demonstrate complementary use of Meta Sampler with CBS (class-balanced sampling) to avoid over-balancing and improve optimization.

Experimental results

Research questions

• RQ1 Can Balanced Softmax correct the bias of standard Softmax caused by imbalanced label distributions?
• RQ2 Does optimizing Balanced Softmax cross-entropy bound the generalization error on balanced test sets?
• RQ3 Can a learnable re-sampling strategy (Meta Sampler) further improve performance on extremely imbalanced datasets?
• RQ4 Do BALMS components yield state-of-the-art results across both long-tailed image classification and instance segmentation datasets?

Key findings

• Balanced Softmax provides more stable and balanced marginal likelihood p(y) across imbalance factors than standard Softmax and competing losses.
• BALMS outperforms state-of-the-art end-to-end and decoupled training methods on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, Places-LT, and LVIS.
• Meta Sampler mitigates the over-balancing issue of CBS in highly imbalanced settings and yields additional gains, especially at large imbalance factors.
• On LVIS, BALMS achieves the best average precision (AP) compared to multiple baselines, including balanced Softmax variants and other loss functions.
• Combining Balanced Softmax with Meta Sampler and decoupled training yields the strongest results across several benchmarks.