[Paper Review] Face-GPS: A Comprehensive Technique for Quantifying Facial Muscle Dynamics in Videos

Juni Kim, Zhikang Dong+1 more|arXiv (Cornell University)|Jan 11, 2024

Facial Nerve Paralysis Treatment and ResearchMedicine3 citations

TL;DR

Face-GPS introduces a novel, explainable method for quantifying facial muscle dynamics in standard video recordings using differential geometry, kernel smoothing, and spectral analysis. By mapping facial landmarks to a canonical face manifold and applying multi-kernel smoothing to optical flow, it achieves 86.1% accuracy in classifying facial emotions on the CK+ dataset, offering a non-invasive, accessible alternative to fEMG and black-box deep learning models.

ABSTRACT

We introduce a novel method that combines differential geometry, kernels smoothing, and spectral analysis to quantify facial muscle activity from widely accessible video recordings, such as those captured on personal smartphones. Our approach emphasizes practicality and accessibility. It has significant potential for applications in national security and plastic surgery. Additionally, it offers remote diagnosis and monitoring for medical conditions such as stroke, Bell's palsy, and acoustic neuroma. Moreover, it is adept at detecting and classifying emotions, from the overt to the subtle. The proposed face muscle analysis technique is an explainable alternative to deep learning methods and a non-invasive substitute to facial electromyography (fEMG).

Motivation & Objective

• To develop a practical, accessible method for quantifying facial muscle activity from standard video recordings, such as those from smartphones.
• To overcome limitations of fEMG and FACS by enabling non-invasive, remote, and scalable facial muscle analysis.
• To enhance interpretability of deep learning-based facial recognition by integrating kernel smoothing with optical flow on a canonical face manifold.
• To enable accurate, automated classification of facial expressions and muscle movements for clinical and security applications.
• To provide a robust, noise-filtering framework that mitigates confounding effects from head movement and background variation.

Proposed method

• The method extracts facial landmarks using MediaPipe and maps them to a canonical face manifold to normalize for head motion and background variation.
• It constructs a triangulated face manifold with 854 triangles and 3,681 landmarks to ensure dense coverage of facial muscle regions.
• Optical flow is computed between consecutive frames to track pixel displacements, which are then smoothed using Gaussian RBF kernels in a multi-kernel smoothing (MKS) framework.
• The MKS approach combines weighted kernel responses from facial muscle descriptors, where weights are derived from a pre-trained FAN model for expression recognition.
• Displacement vectors are transformed back to Cartesian coordinates via inverse affine transformation to quantify muscle movement magnitude and direction.
• The final output is a dynamic vector field (heatmap-like) superimposed on the video, visualizing both direction and intensity of muscle motion.

Experimental results

Research questions

• RQ1 Can facial muscle dynamics be accurately quantified from standard video recordings without specialized equipment?
• RQ2 How can optical flow on a canonical face manifold improve robustness to head movement and background noise?
• RQ3 To what extent does multi-kernel smoothing enhance the interpretability and accuracy of facial muscle movement detection?
• RQ4 Can facial muscle displacement features alone achieve high accuracy in emotion classification without visual appearance cues?
• RQ5 How does the integration of deep learning-based feature weights improve the sensitivity to specific facial actions?

Key findings

• Face-GPS achieves 85.0% average classification accuracy on the CK+ dataset using only facial muscle displacement features, without visual appearance data.
• When enhanced with FAN-based kernel weighting, accuracy increases to 86.1%, demonstrating the value of integrating deep learning priors.
• The method successfully identifies specific Action Units corresponding to distinct emotions, such as lip corner pullers in happiness and brow raisers in fear.
• The multi-kernel smoothing approach effectively reduces noise while preserving meaningful muscle movement patterns in video sequences.
• The visual output—a dynamic vector field—accurately represents both magnitude and direction of facial muscle displacement across frames.
• The framework enables remote, non-invasive monitoring of facial muscle activity, suitable for diagnosing conditions like stroke, Bell’s palsy, and acoustic neuroma.