[Paper Review] Equalized Focal Loss for Dense Long-Tailed Object Detection

Bo Li, Yongqiang Yao+5 more|arXiv (Cornell University)|Jan 7, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes Equalized Focal Loss (EFL), a novel loss function that dynamically adjusts the modulating factor in focal loss based on category-specific imbalance degrees to address both foreground-background and long-tailed category imbalances in one-stage object detectors. EFL achieves 29.2% AP on LVIS v1, setting a new state-of-the-art for one-stage long-tailed detection and demonstrating strong generalization on OpenImages.

ABSTRACT

Despite the recent success of long-tailed object detection, almost all long-tailed object detectors are developed based on the two-stage paradigm. In practice, one-stage detectors are more prevalent in the industry because they have a simple and fast pipeline that is easy to deploy. However, in the long-tailed scenario, this line of work has not been explored so far. In this paper, we investigate whether one-stage detectors can perform well in this case. We discover the primary obstacle that prevents one-stage detectors from achieving excellent performance is: categories suffer from different degrees of positive-negative imbalance problems under the long-tailed data distribution. The conventional focal loss balances the training process with the same modulating factor for all categories, thus failing to handle the long-tailed problem. To address this issue, we propose the Equalized Focal Loss (EFL) that rebalances the loss contribution of positive and negative samples of different categories independently according to their imbalance degrees. Specifically, EFL adopts a category-relevant modulating factor which can be adjusted dynamically by the training status of different categories. Extensive experiments conducted on the challenging LVIS v1 benchmark demonstrate the effectiveness of our proposed method. With an end-to-end training pipeline, EFL achieves 29.2% in terms of overall AP and obtains significant performance improvements on rare categories, surpassing all existing state-of-the-art methods. The code is available at https://github.com/ModelTC/EOD.

Motivation & Objective

• To investigate the feasibility and performance of one-stage object detectors in long-tailed scenarios, where such methods have been underexplored despite industrial preference for their efficiency.
• To identify the root cause of poor performance in one-stage detectors under long-tailed data: inconsistent positive-negative imbalance across categories due to data distribution skew.
• To propose a unified loss function that simultaneously mitigates foreground-background imbalance and inter-category imbalance in one-stage detectors.
• To demonstrate that EFL generalizes well across different backbones and datasets, including LVIS v1 and OpenImages, without requiring data distribution priors.

Proposed method

• EFL introduces a category-relevant modulating factor composed of two decoupled dynamic components: a focusing factor and a weighting factor, both adjusted based on each category's training status and imbalance degree.
• The focusing factor increases attention on hard positive samples for rare categories by scaling the focal loss's exponential term according to category-specific imbalance levels.
• The weighting factor amplifies the loss contribution of rare categories to prevent their gradients from being overwhelmed by frequent categories during training.
• The modulating factor is computed dynamically during training, enabling adaptive loss reweighting without requiring pre-computed class frequencies or data statistics.
• EFL is designed as a plug-in replacement for standard focal loss, compatible with any one-stage detector architecture.
• The method is evaluated using an improved baseline with ResNet-50 on COCO and LVIS v1, using standard training schedules and samplers to validate distribution-agnostic performance.

Experimental results

Research questions

• RQ1 Can one-stage object detectors achieve state-of-the-art performance in long-tailed object detection, despite being underexplored compared to two-stage methods?
• RQ2 Why do standard focal loss and existing reweighting methods fail to generalize effectively in one-stage detectors under long-tailed data distributions?
• RQ3 How can a single loss function simultaneously address both foreground-background imbalance and inter-category imbalance in one-stage detectors?
• RQ4 To what extent does the proposed EFL improve performance on rare categories without degrading performance on frequent ones?
• RQ5 Does EFL maintain strong performance across different data distributions and training protocols, including balanced and long-tailed settings?

Key findings

• EFL achieves 29.2% AP on the LVIS v1 benchmark, surpassing all existing state-of-the-art methods in long-tailed object detection.
• The method significantly improves performance on rare categories, with a notable gain in AP_s (small objects) and AP_l (large objects), indicating robustness across scale.
• On the COCO dataset with balanced class distribution, EFL achieves comparable performance to standard focal loss (42.3% AP), confirming its compatibility with balanced data.
• The derivative analysis shows that EFL increases gradient contribution on hard samples for rare categories, with steeper gradients as imbalance increases.
• EFL generalizes well to OpenImages, demonstrating strong transferability and robustness across different datasets and data distributions.
• The ablation study confirms that the dynamic, category-specific modulating factor is essential—static or shared factors fail to achieve comparable gains.