[Paper Review] Deep Learning-based Spatio Temporal Facial Feature Visual Speech Recognition

Pangoth Santhosh Kumar, Garika Akshay|arXiv (Cornell University)|Apr 30, 2023

Face recognition and analysis4 citations

TL;DR

This paper proposes a deep learning-based visual speech recognition system that combines spatiotemporal facial features from video with LSTM networks to enable secure, password-based authentication on low-resource devices. The model achieves 96.1% accuracy on the MIRACL-VC1 dataset and demonstrates robustness with as few as 10 positive training videos, effectively mitigating spoofing attacks without language or audio dependency.

ABSTRACT

In low-resource computing contexts, such as smartphones and other tiny devices, Both deep learning and machine learning are being used in a lot of identification systems. as authentication techniques. The transparent, contactless, and non-invasive nature of these face recognition technologies driven by AI has led to their meteoric rise in popularity in recent years. While they are mostly successful, there are still methods to get inside without permission by utilising things like pictures, masks, glasses, etc. In this research, we present an alternate authentication process that makes use of both facial recognition and the individual's distinctive temporal facial feature motions while they speak a password. Because the suggested methodology allows for a password to be specified in any language, it is not limited by language. The suggested model attained an accuracy of 96.1% when tested on the industry-standard MIRACL-VC1 dataset, demonstrating its efficacy as a reliable and powerful solution. In addition to being data-efficient, the suggested technique shows promising outcomes with as little as 10 positive video examples for training the model. The effectiveness of the network's training is further proved via comparisons with other combined facial recognition and lip reading models.

Motivation & Objective

• To address the vulnerability of static face recognition systems to spoofing using photos or masks.
• To develop a biometric authentication system that leverages dynamic facial movements during speech, enhancing security beyond static facial features.
• To create a language-agnostic, audio-free authentication method suitable for low-resource devices like smartphones.
• To ensure high accuracy and robustness under varying lighting and environmental conditions.
• To minimize data requirements while maintaining strong performance, enabling deployment on edge devices.

Proposed method

• A pre-trained VGGFace model extracts deep facial features from video frames, capturing identity-specific spatial patterns.
• The sequence of extracted facial features is fed into a stack of Long Short-Term Memory (LSTM) networks to model temporal dynamics of facial movements during speech.
• The model is trained to classify whether a given video clip corresponds to a genuine user reciting their pre-chosen password, without requiring the password content to be known during training.
• The system is evaluated on the MIRACL-VC1 benchmark dataset and a custom dataset collected from smartphone cameras under diverse real-world conditions.
• Model performance is assessed using metrics including accuracy, specificity, sensitivity, AUC, and Equal Error Rate (EER).
• The architecture is designed to be data-efficient, requiring only 10 positive video samples per user for effective training.

Figure 1: Proposed Architecture built on VGGFace

Experimental results

Research questions

• RQ1 Can a deep learning model that combines facial recognition with temporal facial motion analysis achieve higher security than static face recognition?
• RQ2 How well does the model generalize across varying lighting, background, and video quality conditions in real-world deployment?
• RQ3 To what extent can the system be trained with minimal data while maintaining high accuracy and robustness?
• RQ4 How effective is the model in detecting both same-person and different-person spoofing attacks?
• RQ5 Can the system be made language-agnostic and audio-free while still achieving high performance?

Key findings

• The proposed model achieved 96.1% accuracy on the MIRACL-VC1 dataset, demonstrating strong performance in visual speech recognition.
• The system maintained high specificity (95.9%) and sensitivity (98.8%), indicating strong detection of both genuine users and impostors.
• The Equal Error Rate (EER) was 0.021 on MIRACL-VC1 and 0.030 on the custom dataset, indicating low false acceptance and rejection rates.
• The model achieved an AUC score of 0.990 on MIRACL-VC1 and 0.989 on the custom dataset, reflecting excellent discriminative capability.
• The system remained effective with as few as 10 positive training videos per user, highlighting its data efficiency.
• The model outperformed combined state-of-the-art face recognition and lip reading models in terms of robustness and spoofing resistance.

Figure 2: Data Preprocessing Pipeline