[Paper Review] AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Deepfake Detection of Frontal Face Videos

Sahibzada Adil Shahzad, Ammarah Hashmi+3 more|arXiv (Cornell University)|Nov 5, 2023

Digital Media Forensic Detection4 citations

TL;DR

This paper proposes AV-Lip-Sync+, a state-of-the-art audio-visual deepfake detection model that leverages self-supervised audio-visual representations from AV-HuBERT and ViViT to detect inconsistencies between speech and lip movements. By combining multimodal synchronization features with spatiotemporal facial artifacts, it achieves new state-of-the-art performance on FakeAVCeleb and DeepfakeTIMIT datasets, with AUC scores of 0.9998 (HQ) and 0.9996 (LQ) on DeepfakeTIMIT.

ABSTRACT

Multimodal manipulations (also known as audio-visual deepfakes) make it difficult for unimodal deepfake detectors to detect forgeries in multimedia content. To avoid the spread of false propaganda and fake news, timely detection is crucial. The damage to either modality (i.e., visual or audio) can only be discovered through multimodal models that can exploit both pieces of information simultaneously. However, previous methods mainly adopt unimodal video forensics and use supervised pre-training for forgery detection. This study proposes a new method based on a multimodal self-supervised-learning (SSL) feature extractor to exploit inconsistency between audio and visual modalities for multimodal video forgery detection. We use the transformer-based SSL pre-trained Audio-Visual HuBERT (AV-HuBERT) model as a visual and acoustic feature extractor and a multi-scale temporal convolutional neural network to capture the temporal correlation between the audio and visual modalities. Since AV-HuBERT only extracts visual features from the lip region, we also adopt another transformer-based video model to exploit facial features and capture spatial and temporal artifacts caused during the deepfake generation process. Experimental results show that our model outperforms all existing models and achieves new state-of-the-art performance on the FakeAVCeleb and DeepfakeTIMIT datasets.

Motivation & Objective

• To address the challenge of detecting multimodal deepfakes where both audio and visual modalities are manipulated, which unimodal detectors fail to detect.
• To exploit audio-visual inconsistency—particularly lip-sync errors—as a signature for deepfake detection in frontal face videos.
• To improve robustness against diverse deepfake generation techniques by combining self-supervised audio-visual representations with spatiotemporal facial feature learning.
• To outperform existing state-of-the-art models on benchmark datasets like FakeAVCeleb and DeepfakeTIMIT through joint modeling of audio-visual and facial artifacts.

Proposed method

• The model uses AV-HuBERT as a pre-trained audio-visual self-supervised feature extractor to capture synchronization between speech and lip movements.
• A multi-scale temporal convolutional network (TCN) is applied to model temporal correlations between audio and visual modalities.
• A separate vision transformer (ViViT) is employed to extract spatial and temporal facial features beyond the lip region, enhancing detection of non-lip-based artifacts.
• The system fuses features from AV-HuBERT (lip-level) and ViViT (whole-face) to jointly model audio-visual consistency and facial spatiotemporal anomalies.
• The model is fine-tuned end-to-end or with fixed front-ends, and ablation studies validate the contribution of each component.
• Linear layers are used to project and integrate multimodal features before final classification.

Experimental results

Research questions

• RQ1 Can self-supervised audio-visual representations from AV-HuBERT effectively detect lip-sync inconsistencies in deepfake videos?
• RQ2 Does incorporating whole-face facial features via ViViT improve detection performance when lip regions are minimally manipulated?
• RQ3 How does the proposed fusion of audio-visual synchronization and spatiotemporal facial artifacts compare to existing unimodal and multimodal deepfake detectors?
• RQ4 To what extent does AV-Lip-Sync+ generalize across diverse deepfake generation methods on benchmark datasets?

Key findings

• AV-Lip-Sync+ achieves a state-of-the-art AUC of 0.9998 on the High-Quality (HQ) version of the DeepfakeTIMIT dataset, outperforming all prior models.
• On the Low-Quality (LQ) version of DeepfakeTIMIT, the model achieves an AUC of 0.9996, demonstrating robustness to visual degradation.
• The model with joint training of feature extractors (AV-Lip-Sync+ with FE) achieves an AUC of 0.9996 (LQ) and 0.9998 (HQ), indicating strong generalization.
• The ablation study confirms that fixed front-ends with fine-tuned linear layers yield superior performance, suggesting effective feature adaptation.
• On the FakeAVCeleb dataset, AV-Lip-Sync+ achieves new state-of-the-art performance, with AUC scores exceeding 0.96 across all evaluation settings.
• The integration of ViViT for whole-face feature extraction significantly improves detection beyond lip-region analysis, especially in cases of subtle or non-lip manipulation.