[Paper Review] Rethinking Class Imbalance in Machine Learning

Ou Wu|arXiv (Cornell University)|May 6, 2023

Imbalanced Data Classification Techniques4 citations

TL;DR

This paper proposes a novel taxonomy of class imbalance in machine learning, identifying four new types—variance, distance, neighborhood, and quality imbalance—beyond traditional proportion imbalance. It introduces MetaLDA, a meta-learned logit perturbation loss that jointly addresses proportion, variance, and distance imbalances, significantly improving fairness and generalization, with state-of-the-art performance on benchmark datasets like CIFAR100 and CIFAR100-LT.

ABSTRACT

Imbalance learning is a subfield of machine learning that focuses on learning tasks in the presence of class imbalance. Nearly all existing studies refer to class imbalance as a proportion imbalance, where the proportion of training samples in each class is not balanced. The ignorance of the proportion imbalance will result in unfairness between/among classes and poor generalization capability. Previous literature has presented numerous methods for either theoretical/empirical analysis or new methods for imbalance learning. This study presents a new taxonomy of class imbalance in machine learning with a broader scope. Four other types of imbalance, namely, variance, distance, neighborhood, and quality imbalances between/among classes, which may exist in machine learning tasks, are summarized. Two different levels of imbalance including global and local are also presented. Theoretical analysis is used to illustrate the significant impact of the new imbalance types on learning fairness. Moreover, our taxonomy and theoretical conclusions are used to analyze the shortcomings of several classical methods. As an example, we propose a new logit perturbation-based imbalance learning loss when proportion, variance, and distance imbalances exist simultaneously. Several classical losses become the special case of our proposed method. Meta learning is utilized to infer the hyper-parameters related to the three types of imbalance. Experimental results on several benchmark corpora validate the effectiveness of the proposed method.

Motivation & Objective

• To expand the scope of imbalance learning beyond proportion imbalance by identifying and formalizing four new types of class imbalance: variance, distance, neighborhood, and quality imbalance.
• To analyze the theoretical impact of variance and distance imbalances on model fairness and generalization, even in the absence of proportion imbalance.
• To develop a unified learning method that addresses multiple imbalance types simultaneously, improving fairness and performance in long-tailed learning scenarios.
• To validate the effectiveness of the proposed method through extensive experiments on benchmark datasets, demonstrating superiority over classical approaches.

Proposed method

• Proposes a new taxonomy of class imbalance, introducing four new types: variance imbalance (differences in feature variance across classes), distance imbalance (differences in inter-class distances), neighborhood imbalance (differences in local neighborhood structure), and quality imbalance (differences in sample quality or difficulty).
• Introduces a dual-level perspective: global imbalance (overall class distribution) and local imbalance (within-class or local region disparities).
• Develops MetaLDA, a meta-learned logit perturbation loss that incorporates three terms: a proportion-based term (log(π_y / π_c)), a variance-adjusting term (△w_y^T Σ_y △w_y / ||△w_y||²), and a distance-adjusting term (log(Δ̄ / Δ_yc)).
• Uses meta-learning to automatically infer hyperparameters for the three imbalance types, enabling adaptive optimization during training.
• Employs theoretical analysis to demonstrate that variance and distance imbalances alone can cause significant unfairness, even without proportion imbalance.
• Validates the method on CIFAR100 and CIFAR100-LT, showing that each component of the loss contributes to performance gains, with MetaLDA outperforming baseline and classical methods.

Figure 1: The proposed new taxonomy for class imbalance. Existing studies merely deal with proportion imbalance.

Experimental results

Research questions

• RQ1 Can variance and distance imbalances independently cause unfairness and poor generalization, even when class proportions are balanced?
• RQ2 How do multiple types of imbalance—proportion, variance, and distance—interact and jointly affect model performance?
• RQ3 Can a unified loss function be designed to simultaneously mitigate proportion, variance, and distance imbalances?
• RQ4 To what extent do existing classical methods fail when multiple imbalance types co-occur, and why?
• RQ5 Can meta-learning effectively infer optimal hyperparameters for different imbalance types in a data-driven way?

Key findings

• Theoretical analysis confirms that variance and distance imbalances can cause equivalent levels of unfairness as proportion imbalance, even in the absence of class proportion disparity.
• On CIFAR100-LT (100:1), MetaLDA achieves a Top-1 error of 51.55% ± 0.30%, outperforming the baseline (61.23%) and other ablation variants.
• On CIFAR100, MetaLDA achieves a Top-1 error of 16.49% ± 0.17%, surpassing the baseline (18.53%) and all ablation components.
• The ablation study shows that each term in MetaLDA (proportion, variance, distance) contributes independently to performance, with the full method yielding the best results.
• The normalized ISDA (NISDA) variant, which isolates the variance term, outperforms standard ISDA on six datasets, confirming the importance of variance imbalance correction.
• Visualization of perturbation terms shows that variance and distance imbalances are significantly reduced over training epochs, confirming the method's effectiveness in balancing these factors.

Figure 2: Illustrative examples for the four types of imbalance. In (a), the two classes have different proportions of samples; in (b), the two classes have different co-variance matrices; in (c), the middle class has the smallest average inter-class distance; in (d), the two classes have different