[Paper Review] BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition

Boyan Zhou, Quan Cui+2 more|arXiv (Cornell University)|Dec 5, 2019

Domain Adaptation and Few-Shot LearningComputer Science24 references79 citations

TL;DR

BBN presents a Bilateral-Branch Network with a cumulative learning strategy to jointly improve representation and classifier learning for long-tailed visual recognition, showing state-of-the-art performance on several benchmarks.

ABSTRACT

Our work focuses on tackling the challenging but natural visual recognition task of long-tailed data distribution (i.e., a few classes occupy most of the data, while most classes have rarely few samples). In the literature, class re-balancing strategies (e.g., re-weighting and re-sampling) are the prominent and effective methods proposed to alleviate the extreme imbalance for dealing with long-tailed problems. In this paper, we firstly discover that these re-balancing methods achieving satisfactory recognition accuracy owe to that they could significantly promote the classifier learning of deep networks. However, at the same time, they will unexpectedly damage the representative ability of the learned deep features to some extent. Therefore, we propose a unified Bilateral-Branch Network (BBN) to take care of both representation learning and classifier learning simultaneously, where each branch does perform its own duty separately. In particular, our BBN model is further equipped with a novel cumulative learning strategy, which is designed to first learn the universal patterns and then pay attention to the tail data gradually. Extensive experiments on four benchmark datasets, including the large-scale iNaturalist ones, justify that the proposed BBN can significantly outperform state-of-the-art methods. Furthermore, validation experiments can demonstrate both our preliminary discovery and effectiveness of tailored designs in BBN for long-tailed problems. Our method won the first place in the iNaturalist 2019 large scale species classification competition, and our code is open-source and available at https://github.com/Megvii-Nanjing/BBN.

Motivation & Objective

• Investigate how class re-balancing affects representation vs. classifier learning under long-tailed distributions.
• Propose a unified Bilateral-Branch Network (BBN) to jointly optimize representation and classifier learning.
• Develop a cumulative learning strategy to shift focus from universal features to tail data during training.
• Demonstrate effectiveness of BBN on multiple long-tailed benchmarks, including large-scale iNaturalist.

Proposed method

• Two-branch architecture with shared weights: conventional learning branch (uniform sampler) for universal representation, and re-balancing branch (reversed sampler) for tail-focused classifier learning.
• Adaptive fusion of branch outputs via an adaptive parameter alpha that trades off branch contributions.
• Cumulative learning strategy where alpha is a function of training epoch to gradually emphasize tail data.
• End-to-end training with a weighted cross-entropy loss combining both branches’ predictions.
• Inference averages contributions from both branches with alpha fixed at 0.5 for balanced final predictions.

Experimental results

Research questions

• RQ1 How do class re-balancing strategies simultaneously affect feature representation and classifier learning in long-tailed data?
• RQ2 Can a bilateral-branch design that separately handles representation and classifier learning outperform single-branch or two-stage approaches?
• RQ3 Does a cumulative learning schedule that shifts focus from universal features to tail data yield stronger long-tailed recognition performance?
• RQ4 Is sharing backbone weights between branches advantageous for efficiency and performance?

Key findings

• BBN consistently improves over state-of-the-art methods on long-tailed CIFAR-10/100 across imbalance factors.
• BBN also outperforms baselines on large-scale iNaturalist datasets, surpassing strong two-stage fine-tuning methods.
• A reversed sampler for the re-balancing branch yields superior tail-class performance compared to uniform or fully balanced samplers.
• The proposed parabolic decay adaptor for alpha yields the best results among tested strategies.
• Feature representations from the conventional branch remain competitive while the re-balancing branch effectively models tail data, with combined weights yielding balanced classifier behavior.