[Paper Review] Long-tail learning via logit adjustment

Aditya Krishna Menon, Sadeep Jayasumana+4 more|arXiv (Cornell University)|Jul 14, 2020

Imbalanced Data Classification Techniques63 references52 citations

TL;DR

The paper proposes post-hoc logit adjustment and logit-adjusted loss to handle long-tail label distributions, providing Fisher-consistent opportunities to minimize balanced error. It unifies and improves upon prior long-tail methods with strong empirical results.

ABSTRACT

Real-world classification problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels are associated with only a few samples. This poses a challenge for generalisation on such labels, and also makes naïve learning biased towards dominant labels. In this paper, we present two simple modifications of standard softmax cross-entropy training to cope with these challenges. Our techniques revisit the classic idea of logit adjustment based on the label frequencies, either applied post-hoc to a trained model, or enforced in the loss during training. Such adjustment encourages a large relative margin between logits of rare versus dominant labels. These techniques unify and generalise several recent proposals in the literature, while possessing firmer statistical grounding and empirical performance.

Motivation & Objective

• Motivate and address generalisation challenges in long-tail label distributions.
• Introduce two logit-adjustment techniques that bias logits to favor rare labels without sacrificing overall performance.
• Establish Fisher consistency for the balanced error with logit adjustment.
• Demonstrate empirical gains on real-world long-tailed datasets over existing methods.

Proposed method

• Post-hoc logit adjustment adds a label-dependent offset to logits via f_y(x) - tau * log pi_y.
• Logit-adjusted softmax cross-entropy incorporates class priors into the loss, equivalent to g_y(x) = f_y(x) + tau * log pi_y and training a standard scorer.
• The approach yields a pairwise margin loss with Delta_{yy'} = log(pi_{y'}/pi_y) and appropriate alphas, ensuring Fisher consistency for balanced error.
• Comparison shows logit adjustment is additive (not simply weight normalization) and generally more robust across optimizers.
• Provides a general framework extending softmax with a pairwise margin (Eq. 11).
• Implements and evaluates both post-hoc and loss-based logit adjustment across benchmarks.

Experimental results

Research questions

• RQ1 How can logit adjustment modify logits or losses to improve performance on rare classes in long-tailed datasets?
• RQ2 Do post-hoc and training-time logit adjustments align with Fisher consistency for the balanced error?
• RQ3 How do logit-adjusted methods compare to existing post-hoc normalization and loss-modification techniques on standard long-tail benchmarks?
• RQ4 What empirical gains do logit-adjusted methods achieve on CIFAR (LT), ImageNet-LT, and iNaturalist?

Key findings

• Logit adjustment approaches achieve strong improvements over ERM and many baselines on long-tailed datasets.
• Post-hoc logit adjustment often outperforms weight normalization, with further gains when tuning the scaling parameter tau.
• Logit-adjusted loss matches or exceeds post-hoc performance and can yield additional gains.
• On CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist, the logit-adjusted methods consistently reduce balanced error compared to baselines.
• The methods demonstrate robustness across datasets and optimizer choices, supporting the theoretical grounding in Fisher consistency for the balanced error.