[Paper Review] FaceOcc: A Diverse, High-quality Face Occlusion Dataset for Human Face Extraction

Xiangnan Yin, Liming Chen|arXiv (Cornell University)|Jan 20, 2022

Face recognition and analysis5 citations

TL;DR

This paper introduces FaceOcc, a large-scale, high-quality face occlusion dataset comprising over 30,000 diverse, manually annotated occlusions from CelebA-HQ and internet sources, including sunglasses, masks, hands, and scarfs. By combining FaceOcc with CelebAMask-HQ’s attribute masks, the authors train a simple U-Net-based model that achieves SOTA performance in face segmentation, demonstrating that data quality and diversity are more critical than model complexity.

ABSTRACT

Occlusions often occur in face images in the wild, troubling face-related tasks such as landmark detection, 3D reconstruction, and face recognition. It is beneficial to extract face regions from unconstrained face images accurately. However, current face segmentation datasets suffer from small data volumes, few occlusion types, low resolution, and imprecise annotation, limiting the performance of data-driven-based algorithms. This paper proposes a novel face occlusion dataset with manually labeled face occlusions from the CelebA-HQ and the internet. The occlusion types cover sunglasses, spectacles, hands, masks, scarfs, microphones, etc. To the best of our knowledge, it is by far the largest and most comprehensive face occlusion dataset. Combining it with the attribute mask in CelebAMask-HQ, we trained a straightforward face segmentation model but obtained SOTA performance, convincingly demonstrating the effectiveness of the proposed dataset.

Motivation & Objective

• To address the lack of high-quality, diverse face occlusion datasets that hinder accurate face extraction in unconstrained environments.
• To improve face segmentation performance by providing a comprehensive, accurately annotated dataset covering various occlusion types and textures.
• To demonstrate that data quality and diversity can outperform complex model architectures in face extraction tasks.
• To create a scalable, augmentable dataset that supports realistic data augmentation for improved generalization.

Proposed method

• Manual annotation of occlusions—such as sunglasses, masks, hands, and scarfs—on 30,000 high-resolution face images from CelebA-HQ and additional internet-sourced images.
• Integration of occlusion masks with pre-existing attribute masks from CelebAMask-HQ to generate precise face masks via truncated subtraction: $M_{\text{face}} = \max(\hat{M}_{\text{face}} - M_{\text{occ}}, 0)$.
• Application of on-the-fly data augmentation including spatial, color, and occlusion texture transformations to increase data diversity and realism.
• Training of a lightweight U-Net model with ResNet-18 backbone using binary cross-entropy loss and online hard example mining (OHEM).
• Selection of hard samples from FFHQ using the initial model and addition of these to the dataset to improve robustness.
• Use of Apple Pencil and Magic Eraser app for precise manual annotation, ensuring high annotation quality.

Experimental results

Research questions

• RQ1 Can a large, diverse, and accurately annotated face occlusion dataset significantly improve face segmentation performance?
• RQ2 Does data quality and diversity outweigh architectural complexity in training face extraction models?
• RQ3 Can a simple, lightweight model trained on a curated occlusion dataset outperform state-of-the-art methods?
• RQ4 How effective is on-the-fly data augmentation with occlusion texture variation in enhancing model generalization?

Key findings

• The proposed FaceOcc dataset contains over 30,000 images with diverse, manually annotated occlusions, making it the largest and most comprehensive face occlusion dataset to date.
• Training a simple U-Net model with ResNet-18 backbone on FaceOcc and CelebAMask-HQ achieved an IOU of 93.7%, surpassing all prior methods on the COFW benchmark.
• The model achieved a global accuracy of 98.0% and recall of 98.3%, significantly outperforming previous SOTA methods in both metrics.
• Despite a higher input resolution ($256\times256$) than Masi et al.’s ($128\times128$), the model achieved competitive inference speed of 257 FPS.
• The results confirm that high-quality, diverse training data is more impactful than complex model design, as the simple model outperforms prior SOTA methods with no architectural bells and whistles.