[Paper Review] PFLD: A Practical Facial Landmark Detector

Xiaojie Guo, Siyuan Li+6 more|ArXiv.org|Feb 28, 2019

Face recognition and analysis46 references35 citations

TL;DR

PFLD presents a lightweight, end-to-end single-stage facial landmark detector with a geometric-regularized loss and an auxiliary pose branch, achieving high accuracy with ultra-fast mobile speed (over 140 fps) and a tiny 2.1 MB model.

ABSTRACT

Being accurate, efficient, and compact is essential to a facial landmark detector for practical use. To simultaneously consider the three concerns, this paper investigates a neat model with promising detection accuracy under wild environments e.g., unconstrained pose, expression, lighting, and occlusion conditions) and super real-time speed on a mobile device. More concretely, we customize an end-to-end single stage network associated with acceleration techniques. During the training phase, for each sample, rotation information is estimated for geometrically regularizing landmark localization, which is then NOT involved in the testing phase. A novel loss is designed to, besides considering the geometrical regularization, mitigate the issue of data imbalance by adjusting weights of samples to different states, such as large pose, extreme lighting, and occlusion, in the training set. Extensive experiments are conducted to demonstrate the efficacy of our design and reveal its superior performance over state-of-the-art alternatives on widely-adopted challenging benchmarks, i.e., 300W (including iBUG, LFPW, AFW, HELEN, and XM2VTS) and AFLW. Our model can be merely 2.1Mb of size and reach over 140 fps per face on a mobile phone (Qualcomm ARM 845 processor) with high precision, making it attractive for large-scale or real-time applications. We have made our practical system based on PFLD 0.25X model publicly available at \url{http://sites.google.com/view/xjguo/fld} for encouraging comparisons and improvements from the community.

Motivation & Objective

• Address practical needs for accurate, efficient, and compact facial landmark detection in unconstrained environments (pose, lighting, occlusion).
• Develop a single-stage network with speed suitable for mobile/embedded devices.
• Incorporate geometric regularization and data-imbalance handling into training to improve robustness across poses and conditions.
• Enable a highly compact model without sacrificing state-of-the-art accuracy, enabling real-time deployment on mobile hardware.

Proposed method

• Design an end-to-end single-stage backbone network using MobileNet blocks to reduce size and increase speed.
• Introduce an auxiliary network to estimate 3D head pose (yaw, pitch, roll) without requiring extra annotations.
• Propose a novel loss that couples 3D pose-based geometric regularization with sample-wise weighting to address data imbalance.
• Incorporate a multi-scale fully-connected (MS-FC) layer to enlarge the receptive field for precise landmark localization.
• Train with pose-aware augmentation and regularization; testing uses only the backbone network for efficiency.

Experimental results

Research questions

• RQ1 How to achieve high landmark localization accuracy under wild conditions (pose, expression, lighting, occlusion) with a compact model?
• RQ2 Can a geometric-regularized, pose-aware training objective improve robustness over state-of-the-art methods?
• RQ3 Is it possible to maintain real-time performance on mobile devices with a small footprint without sacrificing accuracy?

Key findings

• PFLD variants outperform several state-of-the-art methods on 300W and AFLW benchmarks.
• The 0.25X model achieves strong accuracy with substantially reduced size (2.1 MB) and high speed.
• The 1X model yields even higher accuracy while still enabling real-time performance on mobile hardware.
• The 0.25X and 1X models run at over 140 fps on a Qualcomm ARM 845 processor (per-face).
• The auxiliary pose branch and the proposed loss jointly contribute to improved robustness to pose, occlusion, and lighting variations.