[Paper Review] RefineFace: Refinement Neural Network for High Performance Face Detection

Shifeng Zhang, Cheng Chi+2 more|arXiv (Cornell University)|Sep 10, 2019

Face recognition and analysis80 references20 citations

TL;DR

RefineFace proposes a single-shot refinement neural network that enhances face detection performance by improving regression accuracy and classification recall. It integrates five modules—Selective Two-step Regression, Selective Two-step Classification, Scale-aware Margin Loss, Feature Supervision Module, and Receptive Field Enhancement—achieving state-of-the-art AP of 90.0% on WIDER FACE Hard subset and 37.3 FPS with ResNet-18.

ABSTRACT

Face detection has achieved significant progress in recent years. However, high performance face detection still remains a very challenging problem, especially when there exists many tiny faces. In this paper, we present a single-shot refinement face detector namely RefineFace to achieve high performance. Specifically, it consists of five modules: Selective Two-step Regression (STR), Selective Two-step Classification (STC), Scale-aware Margin Loss (SML), Feature Supervision Module (FSM) and Receptive Field Enhancement (RFE). To enhance the regression ability for high location accuracy, STR coarsely adjusts locations and sizes of anchors from high level detection layers to provide better initialization for subsequent regressor. To improve the classification ability for high recall efficiency, STC first filters out most simple negatives from low level detection layers to reduce search space for subsequent classifier, then SML is applied to better distinguish faces from background at various scales and FSM is introduced to let the backbone learn more discriminative features for classification. Besides, RFE is presented to provide more diverse receptive field to better capture faces in some extreme poses. Extensive experiments conducted on WIDER FACE, AFW, PASCAL Face, FDDB, MAFA demonstrate that our method achieves state-of-the-art results and runs at $37.3$ FPS with ResNet-18 for VGA-resolution images.

Motivation & Objective

• Address the persistent challenge of high-performance face detection, especially for tiny faces and extreme poses.
• Improve location accuracy to reduce false positives due to misaligned bounding boxes (LOC error).
• Enhance recall efficiency to reduce false positives and missed detections (CLS error), particularly in complex scenarios.
• Develop a single-shot detector that maintains real-time inference speed while achieving state-of-the-art accuracy.
• Design a refinement network that improves feature learning and receptive field diversity for robust detection across scales and poses.

Proposed method

• Introduces Selective Two-step Regression (STR) to coarsely adjust anchor locations and sizes from high-level feature maps, providing better initialization for the final regressor.
• Employs Selective Two-step Classification (STC) to filter out easy negative samples from low-level features, reducing search space and improving classification efficiency.
• Applies Scale-aware Margin Loss (SML) to enhance discrimination between faces and background across multiple scales, improving generalization.
• Incorporates Feature Supervision Module (FSM) to guide the backbone network in learning more discriminative features for improved classification.
• Proposes Receptive Field Enhancement (RFE) to diversify receptive fields, enabling better detection of faces in extreme poses.
• Combines all modules into a unified single-shot face detector that jointly optimizes regression and classification with minimal inference overhead.

Experimental results

Research questions

• RQ1 Can a refinement-based single-shot detector achieve state-of-the-art performance on challenging face detection benchmarks like WIDER FACE and FDDB?
• RQ2 How effective is a two-stage refinement strategy in improving regression accuracy without increasing inference latency?
• RQ3 To what extent can selective classification and scale-aware loss improve recall efficiency and reduce false positives?
• RQ4 Can enhanced receptive fields improve detection robustness in extreme pose and occlusion scenarios?
• RQ5 Does the integration of multiple refinement modules lead to consistent performance gains across diverse datasets and face scales?

Key findings

• RefineFace achieves a state-of-the-art average precision (AP) of 90.0% on the WIDER FACE Hard subset, surpassing previous methods by 1.55% over the second-best method STN.
• On the FDDB dataset, RefineFace achieves a true positive rate of 99.11% at 1,000 false positives, setting a new SOTA result under unconstrained conditions.
• On the MAFA dataset, RefineFace achieves 83.9% AP on the whole subset, 96.2% on the masked subset, and 95.7% on the unignored subset, demonstrating strong robustness to occlusions.
• The model runs at 37.3 FPS on VGA-resolution images using ResNet-18, indicating real-time inference capability on standard GPU hardware.
• Ablation studies confirm that each component—STR, STC, SML, FSM, and RFE—contributes significantly to performance gains, especially in reducing both LOC and CLS errors.
• The method outperforms nine state-of-the-art methods and three commercial detectors on PASCAL Face, achieving 99.45% AP, the highest reported score on this benchmark.