[Paper Review] Contrastive Learning based Hybrid Networks for Long-Tailed Image Classification

Peng Wang, Kai Han+3 more|arXiv (Cornell University)|Mar 26, 2021

Domain Adaptation and Few-Shot LearningComputer Science34 references17 citations

TL;DR

This paper proposes a hybrid deep learning framework that combines supervised contrastive learning with cross-entropy loss for long-tailed image classification. By progressively transitioning from feature learning to classifier learning via a curriculum, the method improves feature compactness and inter-class separability, achieving state-of-the-art accuracy on long-tailed datasets, especially under high imbalance ratios.

ABSTRACT

Learning discriminative image representations plays a vital role in long-tailed image classification because it can ease the classifier learning in imbalanced cases. Given the promising performance contrastive learning has shown recently in representation learning, in this work, we explore effective supervised contrastive learning strategies and tailor them to learn better image representations from imbalanced data in order to boost the classification accuracy thereon. Specifically, we propose a novel hybrid network structure being composed of a supervised contrastive loss to learn image representations and a cross-entropy loss to learn classifiers, where the learning is progressively transited from feature learning to the classifier learning to embody the idea that better features make better classifiers. We explore two variants of contrastive loss for feature learning, which vary in the forms but share a common idea of pulling the samples from the same class together in the normalized embedding space and pushing the samples from different classes apart. One of them is the recently proposed supervised contrastive (SC) loss, which is designed on top of the state-of-the-art unsupervised contrastive loss by incorporating positive samples from the same class. The other is a prototypical supervised contrastive (PSC) learning strategy which addresses the intensive memory consumption in standard SC loss and thus shows more promise under limited memory budget. Extensive experiments on three long-tailed classification datasets demonstrate the advantage of the proposed contrastive learning based hybrid networks in long-tailed classification.

Motivation & Objective

• To address the challenge of biased classifiers in long-tailed image classification due to skewed data distributions.
• To investigate whether supervised contrastive learning can produce more discriminative features than cross-entropy in imbalanced settings.
• To design a memory-efficient contrastive learning strategy suitable for limited GPU memory budgets.
• To improve classifier performance by jointly training features and classifiers with a curriculum-based transition from feature learning to classifier learning.
• To demonstrate that better features lead to better classifiers in long-tailed scenarios.

Proposed method

• Introduces a hybrid network that jointly optimizes a supervised contrastive (SC) loss for feature learning and a cross-entropy loss for classifier learning.
• Employs a curriculum learning strategy that progressively shifts training from feature learning (dominated by SC loss) to classifier learning (dominated by cross-entropy loss).
• Proposes a prototypical supervised contrastive (PSC) loss that replaces explicit positive/negative sampling with class prototypes, reducing memory consumption.
• Uses ℓ₂-normalized features and prototypes to decouple feature norms from class frequency, improving stability.
• Derives gradients for PSC loss that are constant for positive-class affinities and prominent for negative-class affinities, enhancing inter-class separability.
• Applies the PSC loss to mitigate overfitting in over-sampled tail classes while maintaining strong discriminative capacity.

Experimental results

Research questions

• RQ1 Can supervised contrastive learning produce more compact and separable features than cross-entropy in long-tailed image classification?
• RQ2 Does a curriculum-based joint training strategy outperform two-stage training in terms of classifier accuracy?
• RQ3 Can a prototypical contrastive loss reduce memory consumption without sacrificing feature quality in long-tailed settings?
• RQ4 Is the PSC loss more effective than standard SC loss under limited GPU memory budgets?
• RQ5 Does improved feature learning via contrastive loss lead to significantly better downstream classification performance?

Key findings

• The proposed hybrid network with curriculum learning achieves 63.05% accuracy on long-tailed CIFAR-100 with an imbalance ratio of 10, significantly outperforming the baseline CE-CE method (59.14%).
• The hybrid-SC model achieves 46.72% accuracy on long-tailed CIFAR-100 (imbalance ratio 10), surpassing the two-stage SC baseline (42.73%) and the hybrid-SC without curriculum (42.58%).
• The PSC variant achieves 44.91% accuracy on long-tailed CIFAR-100 (imbalance ratio 10), demonstrating that PSC is a viable alternative to SC under memory constraints.
• Replacing the contrastive loss with cross-entropy in the hybrid framework leads to a drop in accuracy from 63.05% to 59.14%, proving that contrastive learning is essential for effective feature learning in imbalanced data.
• The curriculum-based joint training strategy significantly improves performance over both two-stage training and fixed-weighted joint training, confirming the value of progressive learning.
• The PSC loss achieves prominent gradients for negative-class prototypes and constant gradients for positive-class affinities, which helps reduce overfitting and enhances inter-class separability.