[Paper Review] An Audio-Visual Attention Based Multimodal Network for Fake Talking Face Videos Detection

Ganglai Wang, Peng Zhang+4 more|arXiv (Cornell University)|Mar 10, 2022

Face recognition and analysis4 citations

TL;DR

This paper proposes FTFDNet, a multimodal deep learning framework that fuses audio and visual features using a novel audio-visual attention mechanism (AVAM) to detect fake talking face videos. Inspired by human multisensory perception, AVAM enhances discriminative feature learning by focusing on lip and cheek regions, achieving 97.00% detection accuracy on the FTFDD dataset.

ABSTRACT

DeepFake based digital facial forgery is threatening the public media security, especially when lip manipulation has been used in talking face generation, the difficulty of fake video detection is further improved. By only changing lip shape to match the given speech, the facial features of identity is hard to be discriminated in such fake talking face videos. Together with the lack of attention on audio stream as the prior knowledge, the detection failure of fake talking face generation also becomes inevitable. Inspired by the decision-making mechanism of human multisensory perception system, which enables the auditory information to enhance post-sensory visual evidence for informed decisions output, in this study, a fake talking face detection framework FTFDNet is proposed by incorporating audio and visual representation to achieve more accurate fake talking face videos detection. Furthermore, an audio-visual attention mechanism (AVAM) is proposed to discover more informative features, which can be seamlessly integrated into any audio-visual CNN architectures by modularization. With the additional AVAM, the proposed FTFDNet is able to achieve a better detection performance on the established dataset (FTFDD). The evaluation of the proposed work has shown an excellent performance on the detection of fake talking face videos, which is able to arrive at a detection rate above 97%.

Motivation & Objective

• Address the challenge of detecting fake talking face videos where only lip movements are manipulated to match speech, making identity features hard to distinguish.
• Overcome the limitation of existing methods that neglect audio streams despite their relevance to lip-synced forgery.
• Leverage human multisensory perception as inspiration to improve detection by fusing audio and visual modalities effectively.
• Develop a modular, plug-in audio-visual attention mechanism (AVAM) that enhances feature representation without modifying backbone architectures.
• Achieve state-of-the-art performance on a newly established benchmark dataset (FTFDD) for fake talking face video detection.

Proposed method

• Propose FTFDNet, a deep neural network that jointly encodes visual and audio features from video frames and audio clips.
• Design AVAM, a novel attention mechanism that models inter-spatial relationships between visual features while incorporating audio modality to guide attention.
• AVAM selectively highlights critical facial regions—especially lips and cheeks—where manipulation is most likely to occur.
• The attention mechanism is implemented as a modular component, allowing seamless integration into any CNN-based audio-visual architecture.
• Train the model end-to-end using binary cross-entropy loss on the FTFDD dataset, with temporal context modeled via frame buffers of size T=3.
• Conduct ablation studies to validate the effectiveness of AVAM versus conventional attention (e.g., CBAM) and optimal frame buffer size.

Experimental results

Research questions

• RQ1 Can audio-visual fusion improve the detection of fake talking face videos where only lip movements are altered?
• RQ2 Does incorporating audio as prior knowledge enhance the model’s ability to detect subtle facial manipulations in lip-synced videos?
• RQ3 How does the proposed AVAM mechanism compare to existing attention modules like CBAM in detecting fake talking face frames?
• RQ4 What is the optimal temporal context length (T) for effective fake video detection using frame buffers?
• RQ5 To what extent does audio-guided attention improve localization of tampered facial regions such as lips and cheeks?

Key findings

• The proposed FTFDNet-AVAM achieves a detection accuracy of 97.00% on the FTFDD dataset, outperforming both the visual-only FTFDNet (96.30%) and FTFDNet-CBAM (96.73%).
• AVAM significantly improves detection performance by focusing attention on lip and cheek regions, which are most vulnerable to manipulation in talking face generation.
• The ablation study confirms that T=3 provides the optimal balance between temporal context and model complexity, as T=5 yields lower accuracy (95.59%) than T=3 (96.30%).
• Audio-visual fusion via AVAM reduces the binary cross-entropy loss to 0.0865, demonstrating better optimization and generalization compared to FTFDNet (0.0933) and FTFDNet-CBAM (0.099).
• Visualization of attention maps confirms that AVAM enhances focus on manipulated facial regions, validating its role in improving feature discrimination.
• The channel attention component of CBAM contributes minimally to performance, indicating that spatial attention guided by audio is more critical than channel-wise modulation in this task.