[Paper Review] Deepfake Video Detection Using Generative Convolutional Vision Transformer

Deressa Wodajo, Solomon Atnafu+1 more|arXiv (Cornell University)|Jul 13, 2023

Digital Media Forensic Detection4 citations

TL;DR

This paper proposes GenConViT, a novel deepfake video detection model that combines ConvNeXt and Swin Transformer for visual feature extraction with Autoencoder and Variational Autoencoder to model latent data distributions. The model achieves state-of-the-art performance with a 95.8% average accuracy and 99.3% AUC across multiple benchmarks, demonstrating strong generalization and robustness in detecting diverse deepfake videos.

ABSTRACT

Deepfakes have raised significant concerns due to their potential to spread false information and compromise digital media integrity. In this work, we propose a Generative Convolutional Vision Transformer (GenConViT) for deepfake video detection. Our model combines ConvNeXt and Swin Transformer models for feature extraction, and it utilizes Autoencoder and Variational Autoencoder to learn from the latent data distribution. By learning from the visual artifacts and latent data distribution, GenConViT achieves improved performance in detecting a wide range of deepfake videos. The model is trained and evaluated on DFDC, FF++, DeepfakeTIMIT, and Celeb-DF v2 datasets, achieving high classification accuracy, F1 scores, and AUC values. The proposed GenConViT model demonstrates robust performance in deepfake video detection, with an average accuracy of 95.8% and an AUC value of 99.3% across the tested datasets. Our proposed model addresses the challenge of generalizability in deepfake detection by leveraging visual and latent features and providing an effective solution for identifying a wide range of fake videos while preserving media integrity. The code for GenConViT is available at https://github.com/erprogs/GenConViT.

Motivation & Objective

• Address the challenge of generalizability in deepfake detection across diverse video distributions.
• Overcome limitations of existing methods that rely solely on visual artifacts and fail on unseen deepfake types.
• Improve detection robustness by jointly modeling visual features and latent data distributions.
• Develop a unified architecture that effectively captures both local and global spatiotemporal patterns in video sequences.
• Provide a reliable solution for preserving digital media integrity in real-world applications.

Proposed method

• Integrate ConvNeXt and Swin Transformer to extract hierarchical local and global visual features from video frames.
• Employ Autoencoder (AE) and Variational Autoencoder (VAE) to learn the underlying latent data distribution of real videos.
• Use the reconstructed outputs from AE and VAE to compare with original frames, identifying discrepancies indicative of manipulation.
• Fuse visual features from ConvNeXt and Swin Transformer with reconstruction errors from AE and VAE for final classification.
• Train the end-to-end model on multiple deepfake datasets using cross-entropy loss and optimize using AdamW.
• Apply data augmentation and class balancing techniques to improve model generalization and prevent overfitting.

Figure 1 : The Proposed Generative Convolutional Vision Transformer (GenConViT) Deepfake Detection Framework.

Experimental results

Research questions

• RQ1 Can combining visual feature extraction with latent distribution modeling improve deepfake detection generalization?
• RQ2 How does GenConViT perform across diverse deepfake datasets such as DFDC, FF++, DeepfakeTIMIT, and Celeb-DF (v2)?
• RQ3 To what extent does modeling both visual artifacts and internal data representations enhance detection robustness?
• RQ4 How does GenConViT compare to state-of-the-art models in terms of accuracy, AUC, and F1-score across benchmark datasets?
• RQ5 Does the fusion of ConvNeXt and Swin Transformer with generative models yield better performance than standalone architectures?

Key findings

• GenConViT achieved a 98.5% accuracy and 99.9% AUC on the DFDC dataset, outperforming previous state-of-the-art models.
• On the FF++ dataset, the model reached 97.0% accuracy and 99.6% AUC, with an F1-score of 97.1%, demonstrating strong performance on diverse deepfake types.
• The model maintained high performance on Celeb-DF (v2), achieving 95.5% F1-score and 98.1% AUC, indicating robustness to complex manipulations.
• GenConViT achieved an average accuracy of 95.8% and an average AUC of 99.3% across all tested datasets, confirming its generalization capability.
• GenConViT(B) slightly outperformed GenConViT(A), with a 99.6% AUC on FF++ and 96.8% F1-score, highlighting the effectiveness of architectural refinement.
• The model’s performance was consistently superior across all metrics, especially in AUC and F1-score, indicating strong discrimination between real and fake classes.

(a) Input Sample Images