[Paper Review] NPVForensics: Jointing Non-critical Phonemes and Visemes for Deepfake Detection

Yu Chen, Yang Yu+3 more|arXiv (Cornell University)|Jun 12, 2023

Speech and Audio Processing4 citations

TL;DR

NPVForensics proposes a novel deepfake detection method that exploits inconsistencies in non-critical phoneme-viseme pairs—regions where advanced forgers often fail to achieve perfect audio-visual calibration. By leveraging a Swin Transformer-based Local Feature Aggregation block, a phoneme-viseme awareness module for cross-modal fusion, co-covariate guided representation alignment, and self-supervised pre-training on large-scale real videos, the method achieves state-of-the-art performance with a 96.9% AUC on the FF++ dataset, demonstrating strong robustness and generalization across datasets.

ABSTRACT

Deepfake technologies empowered by deep learning are rapidly evolving, creating new security concerns for society. Existing multimodal detection methods usually capture audio-visual inconsistencies to expose Deepfake videos. More seriously, the advanced Deepfake technology realizes the audio-visual calibration of the critical phoneme-viseme regions, achieving a more realistic tampering effect, which brings new challenges. To address this problem, we propose a novel Deepfake detection method to mine the correlation between Non-critical Phonemes and Visemes, termed NPVForensics. Firstly, we propose the Local Feature Aggregation block with Swin Transformer (LFA-ST) to construct non-critical phoneme-viseme and corresponding facial feature streams effectively. Secondly, we design a loss function for the fine-grained motion of the talking face to measure the evolutionary consistency of non-critical phoneme-viseme. Next, we design a phoneme-viseme awareness module for cross-modal feature fusion and representation alignment, so that the modality gap can be reduced and the intrinsic complementarity of the two modalities can be better explored. Finally, a self-supervised pre-training strategy is leveraged to thoroughly learn the audio-visual correspondences in natural videos. In this manner, our model can be easily adapted to the downstream Deepfake datasets with fine-tuning. Extensive experiments on existing benchmarks demonstrate that the proposed approach outperforms state-of-the-art methods.

Motivation & Objective

• To address the challenge of detecting advanced deepfakes that perfectly calibrate critical phoneme-viseme pairs, which evade existing multimodal detection methods.
• To exploit subtle audio-visual inconsistencies in non-critical phoneme-viseme regions—areas where forgers often fail to achieve perfect synchronization.
• To improve cross-modal feature alignment and reduce modality gaps through a novel phoneme-viseme awareness module and co-covariate guided representation alignment.
• To enhance model generalization and robustness by employing self-supervised pre-training on large-scale real video data before fine-tuning on deepfake datasets.

Proposed method

• The Local Feature Aggregation block with Swin Transformer (LFA-ST) is designed to extract and preserve both local and global features from audio and visual streams, emphasizing non-critical phoneme-viseme regions.
• A fine-grained motion consistency loss is introduced to measure evolutionary alignment between non-critical phonemes and their corresponding visemes across video frames.
• The phoneme-viseme awareness module enables dynamic, attention-based fusion of audio and visual features, enhancing intrinsic modality complementarity.
• The co-covariate guided representation alignment (CGRA) module reduces modality gap by aligning features based on shared statistical dependencies between phoneme and viseme representations.
• A self-supervised pre-training strategy is applied to large-scale real talking-face videos to learn robust audio-visual correspondences without manual annotations.
• The model is fine-tuned on downstream deepfake datasets, enabling strong zero-shot and cross-dataset generalization.

Figure 1: This figure displays a challenging Lip-Sync forgery video example. On the left, the first row of video frames is real, while the second row is Deepfake. We observe that the forger has calibrated the phoneme-viseme pairs in critical phoneme regions like ‘m’, ‘b’ and ‘p’. Therefore, the diff

Experimental results

Research questions

• RQ1 Can non-critical phoneme-viseme pairs serve as a reliable cue for detecting advanced deepfakes that perfectly calibrate critical phoneme-viseme regions?
• RQ2 How can audio-visual features be effectively fused and aligned to exploit their complementary nature in deepfake detection?
• RQ3 To what extent does self-supervised pre-training on large-scale real videos improve detection performance and generalization on downstream deepfake benchmarks?
• RQ4 What is the contribution of local feature aggregation and fine-grained motion consistency modeling in detecting subtle audio-visual inconsistencies?
• RQ5 How does the proposed cross-modal fusion and alignment mechanism compare to standard multimodal fusion in terms of robustness and accuracy?

Key findings

• NPVForensics achieves a 96.9% AUC on the FF++ dataset, outperforming state-of-the-art methods, with strong cross-dataset generalization.
• The model achieves 96.7% AUC on the FSh dataset and 97.8% on the DFo dataset, demonstrating robustness across diverse deepfake manipulation types.
• Ablation studies show that removing the audio stream reduces performance by 4.6% on FakeAVCeleb, confirming the importance of audio-visual inconsistency detection.
• The co-covariate guided representation alignment (CGRA) module improves AUC by 4.0% when removed, highlighting its role in reducing modality gaps.
• Self-supervised pre-training with 320,000 real video clips outperforms a model pre-trained on 500,000 manually annotated LRW clips, indicating efficiency and scalability of the pre-training approach.
• The LFA-ST module significantly improves performance over ViT and standard Swin Transformer, proving effective in capturing subtle local correlations in non-critical phoneme-viseme regions.

Figure 2: NPVForensics Pipeline. NPVForensics is organized into two stages. The first stage is the pre-training stage. We screen the audio waveform, lip motion stream, and face stream of the real video for non-critical phoneme-viseme, then input them into Swin Transformer Encoder with Local Feature