[Paper Review] Do You Really Mean That? Content Driven Audio-Visual Deepfake Dataset and Multimodal Method for Temporal Forgery Localization

Zhixi Cai, Kalin Stefanov+2 more|arXiv (Cornell University)|Apr 13, 2022

Digital Media Forensic Detection4 citations

TL;DR

This paper introduces the LAV-DF dataset, a large-scale audio-visual deepfake dataset with content-driven manipulations that alter sentiment polarity, and proposes a multimodal 3D CNN method, Boundary Aware Temporal Forgery Detection (BA-TFD), which localizes forged segments using contrastive, boundary matching, and frame classification losses. The method achieves state-of-the-art performance in temporal forgery localization, with 76.90 AP@0.5 and 66.90 AR@100 on the proposed dataset.

ABSTRACT

Due to its high societal impact, deepfake detection is getting active attention in the computer vision community. Most deepfake detection methods rely on identity, facial attributes, and adversarial perturbation-based spatio-temporal modifications at the whole video or random locations while keeping the meaning of the content intact. However, a sophisticated deepfake may contain only a small segment of video/audio manipulation, through which the meaning of the content can be, for example, completely inverted from a sentiment perspective. We introduce a content-driven audio-visual deepfake dataset, termed Localized Audio Visual DeepFake (LAV-DF), explicitly designed for the task of learning temporal forgery localization. Specifically, the content-driven audio-visual manipulations are performed strategically to change the sentiment polarity of the whole video. Our baseline method for benchmarking the proposed dataset is a 3DCNN model, termed as Boundary Aware Temporal Forgery Detection (BA-TFD), which is guided via contrastive, boundary matching, and frame classification loss functions. Our extensive quantitative and qualitative analysis demonstrates the proposed method's strong performance for temporal forgery localization and deepfake detection tasks.

Motivation & Objective

• To address the gap in existing deepfake detection datasets, which assume full-video manipulation, by creating a dataset focused on content-driven, sentiment-altering manipulations.
• To develop a method capable of precisely localizing short, strategically placed forged segments that invert the meaning of real videos.
• To enable research in temporal forgery localization by providing a large-scale, realistic dataset with annotated boundaries of manipulated content.
• To demonstrate that multimodal fusion of audio and video signals significantly improves localization accuracy over unimodal approaches.

Proposed method

• The LAV-DF dataset is constructed by applying content-driven audio-visual reenactment to modify sentiment polarity—e.g., changing 'safe' to 'dangerous'—while preserving identity and naturalness.
• The proposed BA-TFD model uses a 3D CNN backbone with three loss components: contrastive loss for feature alignment, boundary matching loss for precise segment boundary prediction, and frame classification loss for per-frame forgery detection.
• The model is trained end-to-end on the LAV-DF dataset, with supervision on both frame-level and segment-level labels to improve localization accuracy.
• Multimodal fusion is achieved by early concatenation of audio and video features before the 3D convolutional layers, enabling joint modeling of audio-visual cues.
• The method is evaluated using standard metrics: AP@0.5 and AR@100 for temporal localization, and AUC for classification.

Figure 1 : Content-driven audio-visual manipulation. On the left is a real video with the subject saying “Vaccinations are safe”. On the right is an audio-visual deepfake created from the real video based on the change in perceived sentiment where “safe” is changed to “dangerous”. Green-edge and red

Experimental results

Research questions

• RQ1 Can a deepfake detection method effectively localize short, content-altering manipulations that invert sentiment without affecting the rest of the video?
• RQ2 How does multimodal fusion of audio and video improve temporal forgery localization compared to unimodal approaches?
• RQ3 To what extent do contrastive, boundary matching, and frame classification losses contribute to the model’s performance in localization?
• RQ4 How does training on a content-driven dataset improve generalization compared to datasets with random or full-video manipulations?

Key findings

• The proposed BA-TFD method achieves 76.90 AP@0.5 and 66.90 AR@100 on the full LAV-DF dataset, outperforming state-of-the-art methods in temporal forgery localization.
• The visual-only unimodal version of the method achieves 58.55 AP@0.5, while the multimodal version reaches 76.90 AP@0.5, demonstrating the benefit of audio-visual fusion.
• On a subset of the dataset, the visual-only AP@0.5 improves from 58.55 to 83.55, indicating that training on the proposed dataset enhances model generalization.
• The frame classification loss contributes most significantly to performance, followed by boundary matching and contrastive losses, as shown in ablation studies.
• The method achieves 0.990 AUC on deepfake classification, outperforming F3Net (0.520) and MDS (0.828), despite not being trained with a classification loss.

Figure 2 : Generation pipeline of the proposed dataset . The green-edge audio and video frames are the real data, and the red-edge audio and video frames are the generated fake data. The real audio-based transcript is used to decide the location and content to be replaced based on the largest change