[Paper Review] Long-Tailed Recognition by Mutual Information Maximization between Latent Features and Ground-Truth Labels

Min-Kook Suh, Seung‐Woo Seo|arXiv (Cornell University)|May 2, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a novel loss function, Gaussian Mixture Likelihood (GML), that maximizes mutual information between latent features and ground-truth labels to address long-tailed recognition. By modeling class-conditional likelihoods via kernel density estimation with class-wise queues and a teacher-student framework, it seamlessly integrates contrastive learning and logit adjustment, achieving state-of-the-art performance on ImageNet-LT, iNaturalist 2018, CIFAR-LT, and ADE20K semantic segmentation benchmarks.

ABSTRACT

Although contrastive learning methods have shown prevailing performance on a variety of representation learning tasks, they encounter difficulty when the training dataset is long-tailed. Many researchers have combined contrastive learning and a logit adjustment technique to address this problem, but the combinations are done ad-hoc and a theoretical background has not yet been provided. The goal of this paper is to provide the background and further improve the performance. First, we show that the fundamental reason contrastive learning methods struggle with long-tailed tasks is that they try to maximize the mutual information maximization between latent features and input data. As ground-truth labels are not considered in the maximization, they are not able to address imbalances between class labels. Rather, we interpret the long-tailed recognition task as a mutual information maximization between latent features and ground-truth labels. This approach integrates contrastive learning and logit adjustment seamlessly to derive a loss function that shows state-of-the-art performance on long-tailed recognition benchmarks. It also demonstrates its efficacy in image segmentation tasks, verifying its versatility beyond image classification.

Motivation & Objective

• Address the poor performance of contrastive learning on long-tailed datasets due to imbalance in class frequencies.
• Provide a theoretical foundation for combining contrastive learning and logit adjustment, which previous methods lacked.
• Develop a unified framework that integrates contrastive learning and logit adjustment through mutual information maximization with ground-truth labels.
• Verify the method's effectiveness beyond image classification, including in semantic segmentation with imbalanced label distributions.
• Design an efficient, scalable method to model Gaussian mixture likelihoods even for low-frequency tail classes.

Proposed method

• Reframe long-tailed recognition as maximizing mutual information between latent features and ground-truth labels, rather than between features and input data.
• Derive a general loss function, GML, composed of a likelihood term (estimated via Gaussian kernel density estimation) and a logit-adjustment prior term based on class frequency.
• Use multiple class-wise queues instead of a single queue to store contrastive samples, improving representation learning for tail classes.
• Implement a teacher-student distillation strategy where a pre-trained teacher network generates high-quality, up-to-date contrastive samples for training.
• Introduce a learnable temperature parameter τg to adaptively scale logits, improving optimization stability and performance.
• Apply the GML loss to both image classification and semantic segmentation tasks, replacing standard cross-entropy loss with minimal architectural changes.

Experimental results

Research questions

• RQ1 Why do contrastive learning methods fail on long-tailed datasets, despite their success on balanced data?
• RQ2 Can the combination of contrastive learning and logit adjustment be theoretically justified, rather than empirically tuned?
• RQ3 How can mutual information maximization between features and labels be effectively optimized in an imbalanced setting?
• RQ4 Can the proposed method generalize beyond image classification to other tasks like semantic segmentation with long-tailed label distributions?
• RQ5 What is the impact of modeling class-conditional likelihoods via kernel density estimation with class-wise queues and a teacher-student framework?

Key findings

• The proposed GML loss achieves state-of-the-art performance on ImageNet-LT, surpassing previous methods by a significant margin in both top-1 and top-5 accuracy.
• On iNaturalist 2018, the method achieves 85.1% top-1 accuracy, outperforming prior SOTA methods by over 2 percentage points.
• On CIFAR-LT, the method achieves 74.8% top-1 accuracy, demonstrating strong generalization across different long-tailed benchmarks.
• In semantic segmentation on ADE20K, the method improves mIoU from 36.1 (cross-entropy) to 38.1 and mAcc from 45.4 to 51.4, showing strong transferability.
• The optimal hyperparameter α for logit adjustment yields the best mIoU at α=0.2 and best mAcc at α=1.0, confirming the trade-off between head and tail class performance.
• Training the temperature τg leads to a slight but consistent performance gain, improving mAcc from 58.2 to 58.3 on ADE20K.