[Paper Review] Attention Mesh: High-fidelity Face Mesh Prediction in Real-time

Ivan Grishchenko, Artsiom Ablavatski+3 more|arXiv (Cornell University)|Jun 19, 2020

Face recognition and analysisComputer Science8 references51 citations

TL;DR

Attention Mesh is a unified, lightweight model that predicts 3D face mesh with region-focused heads via a spatial transformer, achieving cascaded-level accuracy with faster on-device performance (>50 FPS) on mobile.

ABSTRACT

We present Attention Mesh, a lightweight architecture for 3D face mesh prediction that uses attention to semantically meaningful regions. Our neural network is designed for real-time on-device inference and runs at over 50 FPS on a Pixel 2 phone. Our solution enables applications like AR makeup, eye tracking and AR puppeteering that rely on highly accurate landmarks for eye and lips regions. Our main contribution is a unified network architecture that achieves the same accuracy on facial landmarks as a multi-stage cascaded approach, while being 30 percent faster.

Motivation & Objective

• Motivate high-quality 3D face mesh prediction for on-device AR applications (makeup, puppeteering, eye tracking).
• Eliminate the need for multi-stage cascades by introducing a unified architecture with region-specific heads.
• Achieve high accuracy in salient regions (eyes and lips) while maintaining real-time performance on mobile GPUs.
• Demonstrate trainable attention mechanisms that focus computation on semantically meaningful facial regions.

Proposed method

• Process 256x256 image input to produce a 64x64 feature map.
• Predict all 478 3D face mesh landmarks and crop bounds for regions of interest via a base submodel.
• Use region-specific submodels to predict eyes, irises, and lips from 24x24 attention-cropped features.
• Employ a Spatial Transformer to extract region features from the 64x64 map with an affine transformation θ.
• Train in two phases: (i) independent training with ideal ground-truth crops, (ii) training with crops predicted by the model.
• Optionally include a two-stage inference pipeline that reduces CPU-GPU synchronization by performing the whole pass on GPU.

Experimental results

Research questions

• RQ1 Can a single unified model match the accuracy of a cascaded, region-specific landmark system for high-fidelity face mesh?
• RQ2 Does incorporating region-specific heads with attention improve performance for lips and eyes without sacrificing overall speed?
• RQ3 What are the on-device performance gains when using an end-to-end attention-based mesh on mobile GPUs?
• RQ4 How does the Attention Mesh fare in landmark accuracy across lips, eyes, and iris regions compared to cascaded models?

Key findings

• Attention Mesh runs 25%+ faster than a cascade of separate face and region models on a typical modern mobile device.
• Total inference time on Pixel 2XL: 16.6 ms for the full model; region submodels have 4.18 ms (Lips) and 4.70 ms (Eye & iris).
• Mean normalized 2D landmark error (normalized by 3D interocular distance) for All: 3.11 for Attention Mesh vs. 2.99 (All) for both Mesh and Cascade in aggregate, with Lips: 2.89 (Attention Mesh) vs. 2.70 (Cascade) and Eyes: 6.04 (Attention Mesh) vs. 6.28 (Cascade).
• Attention Mesh achieves comparable lip-region accuracy to cascades and improved performance for eye regions.
• The model supports real-time AR applications such as lipstick rendering and puppeteering with a unified architecture that is easier to train and distribute.