[Paper Review] ResLT: Residual Learning for Long-tailed Recognition

Jiequan Cui, Shu Liu+3 more|arXiv (Cornell University)|Jan 26, 2021

Domain Adaptation and Few-Shot Learning13 citations

TL;DR

This paper proposes ResLT, a novel residual learning framework for long-tailed image recognition by directly re-balancing model parameters across head, medium, and tail classes. It introduces a three-branch residual fusion mechanism—where a main branch handles all classes and two progressively specialized residual branches enhance recognition for medium+tail and tail classes—achieving state-of-the-art accuracy across CIFAR, ImageNet, Places, and iNaturalist benchmarks with consistent improvements over prior methods.

ABSTRACT

Deep learning algorithms face great challenges with long-tailed data distribution which, however, is quite a common case in real-world scenarios. Previous methods tackle the problem from either the aspect of input space (re-sampling classes with different frequencies) or loss space (re-weighting classes with different weights), suffering from heavy over-fitting to tail classes or hard optimization during training. To alleviate these issues, we propose a more fundamental perspective for long-tailed recognition, i.e., from the aspect of parameter space, and aims to preserve specific capacity for classes with low frequencies. From this perspective, the trivial solution utilizes different branches for the head, medium, and tail classes respectively, and then sums their outputs as the final results is not feasible. Instead, we design the effective residual fusion mechanism -- with one main branch optimized to recognize images from all classes, another two residual branches are gradually fused and optimized to enhance images from medium+tail classes and tail classes respectively. Then the branches are aggregated into final results by additive shortcuts. We test our method on several benchmarks, i.e., long-tailed version of CIFAR-10, CIFAR-100, Places, ImageNet, and iNaturalist 2018. Experimental results manifest the effectiveness of our method. Our code is available at https://github.com/jiequancui/ResLT.

Motivation & Objective

• Address the long-tailed recognition problem in real-world datasets where class imbalance severely degrades model performance.
• Overcome limitations of existing input-space (re-sampling) and loss-space (re-weighting) methods that cause overfitting or optimization difficulties.
• Propose a fundamentally new approach by re-balancing model parameters directly, preserving specialized capacity for low-frequency classes.
• Design an effective residual fusion mechanism that enhances tail-class performance without compromising head-class accuracy.

Proposed method

• Introduce a three-branch architecture: one main branch (N_h+m+t) trained on all classes, and two residual branches (N_m+t and N_t) progressively specialized for medium+tail and tail classes respectively.
• Use additive shortcuts to aggregate outputs from all three branches, enabling residual learning that gradually enhances tail-class recognition.
• Maintain parameter specialization by ensuring head, medium, and tail classes still dominate their respective branches during training.
• Train the entire network end-to-end with standard optimization, avoiding complex two-stage training or knowledge distillation.
• Apply the framework to standard CNNs (e.g., ResNet, ResNeXt) with no architectural modifications beyond the residual fusion design.
• Use standard cross-entropy loss with class weights in the main branch, while residual branches learn from the residual error of the main branch.

Experimental results

Research questions

• RQ1 Can direct re-balancing of model parameters across head, medium, and tail classes improve long-tailed recognition performance?
• RQ2 Does a residual fusion mechanism that progressively enhances tail-class recognition outperform standard re-sampling and re-weighting strategies?
• RQ3 Can parameter specialization be preserved in a shared architecture without overfitting to tail classes?
• RQ4 How does the proposed method compare to state-of-the-art one-stage and two-stage methods on diverse long-tailed benchmarks?

Key findings

• On ImageNet-LT, ResLT with ResNeXt-50 achieves 57.6% top-1 accuracy in 180 epochs, surpassing RIDE by 1.6% and SOTA methods by 1.0–1.4%.
• On Places-LT, ResLT achieves 39.8% top-1 accuracy, 1.9% higher than the previous SOTA decoupling method.
• On iNaturalist 2018, ResLT reaches 73.2% top-1 accuracy with ResNet-152, outperforming SOTA by 0.7% under the same training schedule.
• On CIFAR-100-LT with imbalance ratio 0.01, ResLT achieves 49.73% top-1 accuracy with RIDEResNet, surpassing RIDE.
• ResLT maintains a minimal inference cost—only 0.04G FLOPs extra and 17.2ms inference time per batch of 64 images—compared to RIDE.
• Ablation studies confirm that both parameter specialization and the residual learning mechanism are essential for performance gains.