[Paper Review] Adversarial Occlusion-aware Face Detection

Yujia Chen, Lingxiao Song+1 more|arXiv (Cornell University)|Sep 15, 2017

Face recognition and analysis47 references4 citations

TL;DR

This paper proposes Adversarial Occlusion-aware Face Detection (AOFD), a novel face detection framework that jointly detects occluded faces and segments occlusion regions using adversarial training. By generating realistic occlusion patterns and leveraging segmentation as an auxiliary task, AOFD improves detection accuracy on heavily occluded faces, achieving 93.8% AP on MAFA with IoU threshold 0.45 and competitive performance on FDDB.

ABSTRACT

Occluded face detection is a challenging detection task due to the large appearance variations incurred by various real-world occlusions. This paper introduces an Adversarial Occlusion-aware Face Detector (AOFD) by simultaneously detecting occluded faces and segmenting occluded areas. Specifically, we employ an adversarial training strategy to generate occlusion-like face features that are difficult for a face detector to recognize. Occlusion mask is predicted simultaneously while detecting occluded faces and the occluded area is utilized as an auxiliary instead of being regarded as a hindrance. Moreover, the supervisory signals from the segmentation branch will reversely affect the features, aiding in detecting heavily-occluded faces accordingly. Consequently, AOFD is able to find the faces with few exposed facial landmarks with very high confidences and keeps high detection accuracy even for masked faces. Extensive experiments demonstrate that AOFD not only significantly outperforms state-of-the-art methods on the MAFA occluded face detection dataset, but also achieves competitive detection accuracy on benchmark dataset for general face detection such as FDDB.

Motivation & Objective

• Address the challenge of limited real-world occluded face training data in face detection.
• Improve detection performance on heavily occluded faces, especially masked faces, which are common in real-world scenarios.
• Overcome the limitation of traditional methods that treat occlusions as noise or hindrances.
• Explore the effectiveness of using occlusion segmentation as an auxiliary task to enhance feature learning for face detection.
• Develop a robust, end-to-end framework that jointly optimizes face detection and occlusion-aware representation learning.

Proposed method

• Employ a mask generator network to synthesize realistic occlusion patterns on region proposals (RoIs), simulating diverse real-world occlusions such as masks and sunglasses.
• Introduce a compact constraint loss $L_c$ to ensure generated masks are spatially coherent and plausible, avoiding sparse or unrealistic occlusion patterns.
• Implement a multitask learning framework where face detection and occlusion segmentation are trained jointly, with segmentation providing supervisory signals to improve feature learning.
• Use adversarial training to generate hard negative samples (occluded faces) that challenge the detector, increasing model robustness.
• Apply the segmentation branch to predict occluded regions, treating them as informative features rather than noise, thereby improving detection confidence on partially visible faces.
• Optimize the model end-to-end using a combination of detection loss, segmentation loss, and the compact constraint to enhance generalization and feature representation.

Experimental results

Research questions

• RQ1 Can adversarial generation of occlusion patterns effectively augment training data for occluded face detection without relying on real-world annotated occluded images?
• RQ2 Does jointly training a segmentation branch for occlusion regions improve face detection performance, especially on heavily occluded faces?
• RQ3 How does the spatial extent and realism of generated occlusions affect detection accuracy and model robustness?
• RQ4 Can the segmentation of occluded areas serve as a meaningful auxiliary task to enhance feature learning in low-data regimes?
• RQ5 How does the proposed method compare to conventional data augmentation and online hard example mining (OHEM) in terms of performance and training stability?

Key findings

• AOFD achieves 93.8% average precision on the MAFA dataset when the IoU threshold is reduced to 0.45, indicating high detection accuracy even for challenging, partially visible faces.
• With a 1/3 occlusion area on RoIs, AOFD achieves optimal performance, as both too small and too large occlusion areas degrade detection accuracy.
• The compact constraint $L_c$ significantly improves mask realism, increasing average precision from 0.785 to 0.799 when applied, demonstrating its importance in generating plausible occlusions.
• Removing the segmentation branch reduces recall at 1000 FPs on FDDB from 97.88% to 97.13%, confirming its positive impact on feature learning and detection robustness.
• Combining AOFD with online hard example mining (OHEM) yields 81.3% AP on MAFA and 97.88% recall on FDDB, outperforming both individual components, showing complementary benefits.
• The mask generator contributes significantly: removing it reduces AP on MAFA by 3.2% and recall on FDDB by 1.3%, highlighting its role in improving detection confidence on occluded faces.