[Paper Review] Focal Loss for Dense Object Detection

Tsung-Yi Lin, Priya Goyal+3 more|arXiv (Cornell University)|Aug 7, 2017

Advanced Neural Network Applications29 references1,337 citations

TL;DR

Introduces focal loss to address extreme class imbalance in one-stage detectors, enabling RetinaNet to surpass previous state-of-the-art detectors while maintaining speed.

ABSTRACT

The highest accuracy object detectors to date are based on a two-stage approach popularized by R-CNN, where a classifier is applied to a sparse set of candidate object locations. In contrast, one-stage detectors that are applied over a regular, dense sampling of possible object locations have the potential to be faster and simpler, but have trailed the accuracy of two-stage detectors thus far. In this paper, we investigate why this is the case. We discover that the extreme foreground-background class imbalance encountered during training of dense detectors is the central cause. We propose to address this class imbalance by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples. Our novel Focal Loss focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training. To evaluate the effectiveness of our loss, we design and train a simple dense detector we call RetinaNet. Our results show that when trained with the focal loss, RetinaNet is able to match the speed of previous one-stage detectors while surpassing the accuracy of all existing state-of-the-art two-stage detectors. Code is at: https://github.com/facebookresearch/Detectron.

Motivation & Objective

• Identify the primary cause of poor accuracy in dense one-stage detectors.
• Propose a loss function that focuses learning on hard examples to address foreground-background imbalance.
• Design a simple yet effective one-stage detector (RetinaNet) that achieves state-of-the-art accuracy.
• Demonstrate that focal loss enables competitive or superior accuracy-while maintaining fast inference speeds.

Proposed method

• Formalize focal loss as FL(p_t) = -alpha_t (1 - p_t)^gamma log(p_t) and show its equivalence to cross-entropy when gamma=0.
• Introduce an alpha-balanced variant to address class imbalance.
• Initialize training with a foreground probability prior pi to stabilize learning.
• Build RetinaNet with an FPN backbone, a classification subnet, and a box regression subnet for one-stage dense detection.
• Train end-to-end using SGD on ~100k anchors per image, using focal loss for classification and smooth L1 for regression.

Experimental results

Research questions

• RQ1 Can focal loss mitigate extreme foreground-background imbalance in dense one-stage detectors?
• RQ2 Does focusing training on hard examples via focal loss improve COCO AP compared to CE and OHEM baselines?
• RQ3 Is RetinaNet able to match or surpass two-stage detectors in accuracy while preserving speed?
• RQ4 What are the optimal gamma and alpha settings for focal loss in this context?
• RQ5 How do design choices in anchors and feature pyramid networks influence performance when using focal loss?

Key findings

• Focal loss yields substantial AP gains over CE and alpha-balanced CE, with gamma=2 providing strong improvements.
• RetinaNet with ResNet-101-FPN achieves 39.1 AP on COCO test-dev, surpassing prior one-stage and many two-stage methods.
• Compared to OHEM baselines, FL-based training performs better in AP (e.g., FL outperforms OHEM variants by over 3 AP points).
• A simple one-stage detector using focal loss can approach or exceed state-of-the-art two-stage detectors in accuracy while maintaining competitive speed.
• The method remains robust across a range of gamma values and anchor configurations, with best results near gamma=2 and alpha around 0.25.