[Paper Review] 3DPalsyNet: A Facial Palsy Grading and Motion Recognition Framework using Fully 3D Convolutional Neural Networks

Gary Storey, Richard Jiang+3 more|arXiv (Cornell University)|May 31, 2019

Facial Nerve Paralysis Treatment and Research27 references4 citations

TL;DR

3DPalsyNet is a 3D convolutional neural network framework with a ResNet backbone that enables end-to-end facial palsy grading and mouth motion recognition from video sequences. By leveraging transfer learning from Kinetics and combining center loss with softmax loss, it achieves 86% accuracy in motion recognition and 82% in grading, with optimal performance at 8-frame durations.

ABSTRACT

The capability to perform facial analysis from video sequences has significant potential to positively impact in many areas of life. One such area relates to the medical domain to specifically aid in the diagnosis and rehabilitation of patients with facial palsy. With this application in mind, this paper presents an end-to-end framework, named 3DPalsyNet, for the tasks of mouth motion recognition and facial palsy grading. 3DPalsyNet utilizes a 3D CNN architecture with a ResNet backbone for the prediction of these dynamic tasks. Leveraging transfer learning from a 3D CNNs pre-trained on the Kinetics data set for general action recognition, the model is modified to apply joint supervised learning using center and softmax loss concepts. 3DPalsyNet is evaluated on a test set consisting of individuals with varying ranges of facial palsy and mouth motions and the results have shown an attractive level of classification accuracy in these task of 82% and 86% respectively. The frame duration and the loss function affect was studied in terms of the predictive qualities of the proposed 3DPalsyNet, where it was found shorter frame duration's of 8 performed best for this specific task. Centre loss and softmax have shown improvements in spatio-temporal feature learning than softmax loss alone, this is in agreement with earlier work involving the spatial domain.

Motivation & Objective

• To develop an end-to-end framework for automated facial palsy grading and mouth motion recognition using video data.
• To improve spatio-temporal feature learning in facial motion analysis through joint supervised learning with center and softmax loss.
• To evaluate the impact of frame duration and loss function design on classification performance in facial palsy assessment.
• To enable clinical applications in diagnosis and rehabilitation support through robust, deep learning-based video analysis.
• To demonstrate the effectiveness of 3D CNNs with ResNet architecture for dynamic facial expression analysis in medical contexts.

Proposed method

• The framework employs a 3D CNN with a ResNet backbone to extract spatio-temporal features from video clips of facial movements.
• Transfer learning is applied using a 3D CNN pre-trained on the Kinetics dataset for general action recognition.
• Joint supervised learning is implemented using both center loss and softmax loss to enhance feature discrimination.
• The model is trained and evaluated on a diverse test set of individuals with varying degrees of facial palsy and motion types.
• Frame duration is systematically varied (e.g., 8, 16, 32 frames) to assess its impact on predictive performance.
• The architecture is optimized for temporal modeling by preserving sequential dynamics in facial motion sequences.

Experimental results

Research questions

• RQ1 How effective is a 3D CNN with ResNet backbone in recognizing dynamic facial motions associated with facial palsy?
• RQ2 What is the optimal frame duration for accurate classification in facial palsy grading and motion recognition?
• RQ3 How does the combination of center loss and softmax loss improve spatio-temporal feature learning compared to softmax loss alone?
• RQ4 Can transfer learning from Kinetics improve performance on the facial palsy classification task?
• RQ5 How does the model generalize across individuals with varying degrees of facial palsy severity?

Key findings

• The 3DPalsyNet framework achieved a classification accuracy of 86% for mouth motion recognition on the test set.
• Facial palsy grading was performed with an accuracy of 82% using the proposed framework.
• Shorter frame durations of 8 frames yielded the best performance, outperforming longer durations such as 16 or 32 frames.
• The combination of center loss and softmax loss significantly improved spatio-temporal feature learning compared to using softmax loss alone.
• The use of transfer learning from the Kinetics dataset enhanced model generalization and performance on the facial palsy task.
• The results demonstrate the feasibility of using 3D CNNs for automated, clinically relevant facial palsy assessment from video.