[Paper Review] Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs

Yury Kartynnik, Artsiom Ablavatski+2 more|arXiv (Cornell University)|Jul 15, 2019

Face recognition and analysisComputer Science6 references58 citations

TL;DR

The paper presents a neural network model that predicts a dense 468-point 3D facial mesh from a single monocular video frame, optimized for real-time mobile GPU inference with several light-weight variants.

ABSTRACT

We present an end-to-end neural network-based model for inferring an approximate 3D mesh representation of a human face from single camera input for AR applications. The relatively dense mesh model of 468 vertices is well-suited for face-based AR effects. The proposed model demonstrates super-realtime inference speed on mobile GPUs (100-1000+ FPS, depending on the device and model variant) and a high prediction quality that is comparable to the variance in manual annotations of the same image.

Motivation & Objective

• Motivate and enable real-time 3D facial surface geometry capture for AR from a single RGB frame.
• Propose a 468-point facial mesh topology suitable for expressive AR effects.
• Achieve mobile GPU real-time inference with accuracy comparable to inter-annotator variability.
• Provide an end-to-end pipeline from face detection and alignment to mesh prediction and temporal stabilization.

Proposed method

• Use a residual neural network to regress 3D coordinates for 468 mesh vertices from a cropped 256x256 (full) or 128x128 (lighter) input.
• Produce x,y coordinates in image space and z as depth relative to a reference plane, with re-scaling to maintain aspect ratios.
• Incorporate a lightweight face detector and a face alignment step to crop and align input frames.
• Apply a 1D temporal filter (1 Euro filter-inspired) to each landmark coordinate to reduce jitter in video sequences.
• Train using synthetic 3DMM renderings plus 2D semantic landmarks for bootstrap, followed by iterative refinement on in-the-wild data.

Experimental results

Research questions

• RQ1 Can a dense 3D facial mesh (468 points) be inferred from monocular video with high fidelity on mobile hardware?
• RQ2 Does a lightweight architecture suffice for real-time inference across devices with varying GPUs and CPUs?
• RQ3 How does the proposed temporal filtering affect visual stability of landmark trajectories in video?
• RQ4 What is the impact of input resolution on accuracy and speed in mobile contexts?
• RQ5 Can synthetic-plus-refinement training achieve visually plausible depth for AR applications without full metric accuracy?

Key findings

• Full model achieves 3.96% inter-ocular distance (IOD) mean absolute distance (MAD) on 2D coordinates; depth is learned from synthetic supervision.
• GPU-targeted full model runs at 2.5 ms per frame on iPhone XS and 7.4 ms on Pixel 3 for 256x256 input.
• Light model at 128x128 achieves 1 ms (iPhone XS) and 3.4 ms (Pixel 3) with 5.15% IOD MAD.
• Lightest model at 128x128 achieves 0.7 ms (iPhone XS) and 2.6 ms (Pixel 3) with 5.29% IOD MAD.
• Temporal filtering reduces frame-to-frame jitter while preserving responsiveness in video sequences.