[Paper Review] Deepfake Detection Scheme Based on Vision Transformer and Distillation

Young Jin Heo, Young Ju Choi+2 more|arXiv (Cornell University)|Apr 3, 2021

Digital Media Forensic Detection45 references41 citations

TL;DR

The paper proposes a Vision Transformer-based deepfake detector that fuses patch embeddings with CNN features and uses distillation from EfficientNet to improve robustness and accuracy on the DFDC full dataset, achieving higher AUC and F1 than the prior SOTA without ensembling.

ABSTRACT

Deepfake is the manipulated video made with a generative deep learning technique such as Generative Adversarial Networks (GANs) or Auto Encoder that anyone can utilize. Recently, with the increase of Deepfake videos, some classifiers consisting of the convolutional neural network that can distinguish fake videos as well as deepfake datasets have been actively created. However, the previous studies based on the CNN structure have the problem of not only overfitting, but also considerable misjudging fake video as real ones. In this paper, we propose a Vision Transformer model with distillation methodology for detecting fake videos. We design that a CNN features and patch-based positioning model learns to interact with all positions to find the artifact region for solving false negative problem. Through comparative analysis on Deepfake Detection (DFDC) Dataset, we verify that the proposed scheme with patch embedding as input outperforms the state-of-the-art using the combined CNN features. Without ensemble technique, our model obtains 0.978 of AUC and 91.9 of f1 score, while previous SOTA model yields 0.972 of AUC and 90.6 of f1 score on the same condition.

Motivation & Objective

• Motivate robust deepfake detection amid overfitting in CNN-based methods.
• Leverage Vision Transformer to capture global context and artifacts across patches.
• Integrate patch embeddings with CNN features to improve detection of manipulated regions.
• Employ distillation from a strong teacher (EfficientNet-B7) to reduce overfitting and enhance generalization.

Proposed method

• Split input frames into patches and obtain patch embeddings.
• Combine patch embeddings with CNN features from EfficientNet to form input tokens.
• Add class and distillation tokens and process with a Transformer encoder (MSA + GELU MLP).
• Use a distillation mechanism where the teacher model guides the distillation token during training.
• Train with a combined loss that balances BCE on class and distillation outputs (λ = 0.5).
• Use pretraining on DFDC and patch size 32 with embedding dimension 1024; 16 attention heads and 24 Transformer layers.

Experimental results

Research questions

• RQ1 Can Vision Transformer with patch embedding and CNN feature fusion outperform CNN-based detectors on the DFDC full dataset without ensemble methods?
• RQ2 Does distillation from EfficientNet-B7 improve deepfake detection robustness and reduce overfitting for Transformer-based detectors?
• RQ3 How does the proposed method compare to the prior SOTA in AUC and F1 under the same training/testing conditions?

Key findings

• Achieved ROC-AUC of 0.978 on DFDC full dataset, higher than the SOTA 0.972.
• Achieved F1 score of 91.9, higher than the SOTA 90.6 under the same threshold (0.55).
• Model shows robustness on fake videos with better discrimination, as evidenced by lower validation fake loss and comparable real loss.
• Distillation tokens outperform class tokens on testing, contributing to improved detection.