[Paper Review] Class-Balanced Loss Based on Effective Number of Samples

Yin Cui, Menglin Jia+3 more|arXiv (Cornell University)|Jan 16, 2019

Domain Adaptation and Few-Shot Learning46 references128 citations

TL;DR

The paper introduces a class-balanced loss based on the effective number of samples to address long-tailed data, tying sample weighting to an exponential effective sample count and demonstrating gains across CIFAR, iNaturalist, ImageNet, and other datasets.

ABSTRACT

With the rapid increase of large-scale, real-world datasets, it becomes critical to address the problem of long-tailed data distribution (i.e., a few classes account for most of the data, while most classes are under-represented). Existing solutions typically adopt class re-balancing strategies such as re-sampling and re-weighting based on the number of observations for each class. In this work, we argue that as the number of samples increases, the additional benefit of a newly added data point will diminish. We introduce a novel theoretical framework to measure data overlap by associating with each sample a small neighboring region rather than a single point. The effective number of samples is defined as the volume of samples and can be calculated by a simple formula $(1-β^{n})/(1-β)$, where $n$ is the number of samples and $β\in [0,1)$ is a hyperparameter. We design a re-weighting scheme that uses the effective number of samples for each class to re-balance the loss, thereby yielding a class-balanced loss. Comprehensive experiments are conducted on artificially induced long-tailed CIFAR datasets and large-scale datasets including ImageNet and iNaturalist. Our results show that when trained with the proposed class-balanced loss, the network is able to achieve significant performance gains on long-tailed datasets.

Motivation & Objective

• Motivate and model the diminishing returns of additional data due to overlap in real-world long-tailed distributions.
• Define the effective number of samples to quantify data overlap.
• Propose a loss re-weighting term inversely proportional to the effective number of samples per class.
• Show the class-balanced loss can be applied to softmax, sigmoid, and focal losses across datasets.

Proposed method

• Define effective number of samples E_n = (1 - beta^n) / (1 - beta) with beta in [0,1).
• Assume a dataset-wide N and beta = (N-1)/N to compute E_n for each class.
• Introduce class-balanced weights proportional to 1 / E_{n_i}, normalized to sum to C.
• Apply the class-balanced weighting as CB loss: CB = (1 - beta) / (1 - beta^{n_y}) * L(p, y).
• Derive CB versions for softmax cross-entropy, sigmoid cross-entropy, and focal loss (CB_softmax, CB_sigmoid, CB_focal).
• Show CB_focal corresponds to setting alpha_t in focal loss as (1 - beta)/(1 - beta^{n_y}).

Experimental results

Research questions

• RQ1 How can the effective number of samples be defined to capture data overlap in long-tailed distributions?
• RQ2 Can re-weighting by the inverse effective number of samples improve performance over inverse class frequency?
• RQ3 Is the proposed class-balanced loss agnostic to the base loss function and applicable across softmax, sigmoid, and focal losses?
• RQ4 What gains do CB losses yield on artificial long-tailed CIFAR and real-world large-scale datasets like ImageNet and iNaturalist?

Key findings

• CB loss yields significant performance gains on long-tailed datasets across loss functions.
• CB_softmax, CB_sigmoid, and CB_focal improve over their unbalanced counterparts in long-tailed CIFAR experiments.
• CB_focal with beta ~ 0.999 and gamma around 0.5–2.0 achieves strong results on iNaturalist and ImageNet.
• On large-scale data, CB_focal outperforms softmax cross-entropy notably on iNaturalist and matches or surpasses baseline on ImageNet.