[Paper Review] MaskFace: multi-task face and landmark detector

D. A. Yashunin, Tamir Baydasov+1 more|arXiv (Cornell University)|May 19, 2020

Face recognition and analysis60 references13 citations

TL;DR

MaskFace proposes a multi-task face and landmark detection model that integrates a Mask R-CNN-inspired keypoint head into a face detection framework, using RoIAlign for precise feature extraction. By jointly predicting faces and landmarks with minimal computational overhead, it achieves state-of-the-art accuracy on WIDER FACE, AFLW, and 300W datasets, outperforming both single-task and multi-task models.

ABSTRACT

Currently in the domain of facial analysis single task approaches for face detection and landmark localization dominate. In this paper we draw attention to multi-task models solving both tasks simultaneously. We present a highly accurate model for face and landmark detection. The method, called MaskFace, extends previous face detection approaches by adding a keypoint prediction head. The new keypoint head adopts ideas of Mask R-CNN by extracting facial features with a RoIAlign layer. The keypoint head adds small computational overhead in the case of few faces in the image while improving the accuracy dramatically. We evaluate MaskFace's performance on a face detection task on the AFW, PASCAL face, FDDB, WIDER FACE datasets and a landmark localization task on the AFLW, 300W datasets. For both tasks MaskFace achieves state-of-the-art results outperforming many of single-task and multi-task models.

Motivation & Objective

• To address the limitations of sequential single-task face detection and landmark localization pipelines that propagate errors and lack feature sharing.
• To improve accuracy in both face detection and landmark localization by jointly training a unified model with shared representations.
• To explore the effectiveness of a Mask R-CNN-style keypoint head for facial landmark prediction, especially for sparse and dense landmarks.
• To evaluate the impact of model architecture components such as context modules, feature pyramids, and backbone choice on detection and localization performance.
• To demonstrate that a well-designed multi-task model can outperform more complex single-task or multi-task baselines without additional data augmentation or supervision.

Proposed method

• Extends RetinaFace and SSH-style face detectors by adding a keypoint prediction head inspired by Mask R-CNN.
• Uses RoIAlign to extract aligned features from predicted bounding boxes, ensuring pixel-to-pixel correspondence between features and landmarks.
• Employs a feature pyramid network (FPN) to enhance detection of small faces by combining multi-scale features.
• Introduces context modules to expand the receptive field and improve feature representation in shallow layers.
• Applies focal loss and online hard example mining (OHEM) to handle class imbalance in anchor-based face detection.
• Uses a regression-based keypoint head with L1/L2 loss for landmark localization, trained end-to-end with the detection head.

Experimental results

Research questions

• RQ1 Can a Mask R-CNN-style mask head be effectively adapted for facial keypoint detection to improve localization accuracy?
• RQ2 Does joint training of face detection and landmark localization with shared features lead to better performance than single-task models?
• RQ3 How does the inclusion of a keypoint head affect face detection accuracy, and can interference between tasks be mitigated?
• RQ4 What is the impact of backbone architecture, context modules, and feature pyramid on detection and landmark localization performance?
• RQ5 How does MaskFace compare in accuracy and inference speed to state-of-the-art single-task and multi-task models?

Key findings

• MaskFace achieves state-of-the-art AP of 0.9724 on the hard subset of WIDER FACE using ResNeXt-152, outperforming previous models.
• On AFLW, MaskFace achieves a CED@0.95 of 3.39 for 5 facial landmarks, the best result among methods without extra data or supervision.
• The model maintains high efficiency, achieving ~20 FPS on 640x480 images with ResNet-50 and ~11 ms per face for landmark prediction.
• Ablation studies show that context modules and the keypoint head significantly improve detection of hard (tiny/occluded) faces.
• The keypoint head adds negligible computational cost—only 0.11 ms per face—making it suitable for real-time applications.
• ResNeXt-152 backbone yields the best performance (AP: 0.9724, CED: 3.39), confirming that stronger representations boost accuracy.