[Paper Review] The Devil is in Classification: A Simple Framework for Long-tail Object Detection and Instance Segmentation

Tao Wang, Yu Li+6 more|arXiv (Cornell University)|Jul 23, 2020

Advanced Neural Network Applications38 references4 citations

TL;DR

This paper proposes SimCal, a simple yet effective calibration framework to address long-tail instance segmentation by correcting classification head bias in two-stage models like Mask R-CNN. By using bi-level class-balanced sampling to retrain the classification head on underrepresented classes and combining predictions from original and calibrated heads via a novel ensemble strategy, SimCal achieves state-of-the-art performance on LVIS and COCO-LT, significantly boosting AP for tail classes without sacrificing head class performance.

ABSTRACT

Most existing object instance detection and segmentation models only work well on fairly balanced benchmarks where per-category training sample numbers are comparable, such as COCO. They tend to suffer performance drop on realistic datasets that are usually long-tailed. This work aims to study and address such open challenges. Specifically, we systematically investigate performance drop of the state-of-the-art two-stage instance segmentation model Mask R-CNN on the recent long-tail LVIS dataset, and unveil that a major cause is the inaccurate classification of object proposals. Based on such an observation, we first consider various techniques for improving long-tail classification performance which indeed enhance instance segmentation results. We then propose a simple calibration framework to more effectively alleviate classification head bias with a bi-level class balanced sampling approach. Without bells and whistles, it significantly boosts the performance of instance segmentation for tail classes on the recent LVIS dataset and our sampled COCO-LT dataset. Our analysis provides useful insights for solving long-tail instance detection and segmentation problems, and the straightforward \emph{SimCal} method can serve as a simple but strong baseline. With the method we have won the 2019 LVIS challenge. Codes and models are available at https://github.com/twangnh/SimCal.

Motivation & Objective

• To investigate why state-of-the-art two-stage instance segmentation models like Mask R-CNN underperform on long-tailed datasets such as LVIS.
• To identify inaccurate object proposal classification as the primary cause of performance drop on tail classes.
• To develop a simple, effective calibration framework that improves classification head performance without retraining the full model.
• To maintain high performance on head classes while significantly boosting performance on long-tail (tail) classes.
• To provide a strong, practical baseline for long-tail instance segmentation that can be easily integrated into existing models.

Proposed method

• Propose a decoupled learning scheme where the classification head is calibrated after standard training using a bi-level sampling strategy that combines image-level and instance-level sampling to balance class distribution.
• Use a new bi-level sampling method to collect proposal samples that are class-balanced across all categories, especially improving representation of tail classes.
• Retrain only the classification head on the balanced proposal set using standard cross-entropy loss, minimizing interference with detection and mask heads.
• Introduce a dual-head inference strategy that combines predictions from the original and calibrated classification heads using a novel output combination method (sel), which outperforms alternatives like averaging or NMS-based fusion.
• Apply a scaling and normalization strategy (sel-scale, sel-norm) to align confidence scores between original and calibrated heads, improving detection quality.
• Use a simple, fixed learning rate (0.01) during calibration, which matches standard training and avoids the need for extensive hyperparameter tuning.

Experimental results

Research questions

• RQ1 Why do state-of-the-art two-stage instance segmentation models fail on long-tailed datasets like LVIS?
• RQ2 To what extent does classification head bias—caused by imbalanced training samples—contribute to performance degradation on tail classes?
• RQ3 Can a simple, post-training calibration of the classification head significantly improve performance on long-tailed instance segmentation without retraining the entire model?
• RQ4 What is the optimal strategy for combining predictions from the original and calibrated classification heads to maximize performance across all classes?
• RQ5 Does the performance gain from calibration depend on which layers of the classification head are fine-tuned?

Key findings

• The primary cause of performance drop on long-tailed datasets is inaccurate classification of object proposals, not region proposal network (RPN) quality.
• SimCal achieves 23.4 AP on the LVIS dataset using ResNet-50-FPN, a significant improvement over the baseline 18.0 AP, with a 5.4-point gain on tail classes (AP_f from 16.4 to 22.5).
• The proposed combining method (sel) achieves 21.1 AP, outperforming alternatives like averaging (20.3 AP) and NMS-based fusion (19.8 AP).
• Calibrating the full 3-layer classification head yields the best performance (22.2 AP), outperforming partial calibration on last 2 or 1 layers.
• The optimal learning rate for calibration is 0.01, matching standard training, indicating no need for low learning rates typical in fine-tuning.
• The variance in performance is much higher for tail classes (e.g., 1.3 for AP_f), confirming the challenge of long-tail learning, while head classes show lower variance due to abundant training data.