[Paper Review] Deepfake Video Detection with Spatiotemporal Dropout Transformer

Daichi Zhang, Fanzhao Lin+4 more|arXiv (Cornell University)|Jul 14, 2022

Digital Media Forensic Detection4 citations

TL;DR

This paper proposes a patch-level deepfake video detection framework, Spatiotemporal Dropout Transformer (STD-Former), which reorganizes facial video frames into bags of non-overlapping patches and feeds them into a Vision Transformer (ViT) for robust representation learning. A novel spatiotemporal dropout operation—applying random frame and patch removal—enhances model generalization and robustness by exploiting dynamic spatiotemporal inconsistencies in fake videos, achieving state-of-the-art performance on multiple benchmarks with a 11.15% average accuracy gain over ViT backbones.

ABSTRACT

While the abuse of deepfake technology has caused serious concerns recently, how to detect deepfake videos is still a challenge due to the high photo-realistic synthesis of each frame. Existing image-level approaches often focus on single frame and ignore the spatiotemporal cues hidden in deepfake videos, resulting in poor generalization and robustness. The key of a video-level detector is to fully exploit the spatiotemporal inconsistency distributed in local facial regions across different frames in deepfake videos. Inspired by that, this paper proposes a simple yet effective patch-level approach to facilitate deepfake video detection via spatiotemporal dropout transformer. The approach reorganizes each input video into bag of patches that is then fed into a vision transformer to achieve robust representation. Specifically, a spatiotemporal dropout operation is proposed to fully explore patch-level spatiotemporal cues and serve as effective data augmentation to further enhance model's robustness and generalization ability. The operation is flexible and can be easily plugged into existing vision transformers. Extensive experiments demonstrate the effectiveness of our approach against 25 state-of-the-arts with impressive robustness, generalizability, and representation ability.

Motivation & Objective

• To address the limitations of image-level deepfake detection methods that ignore temporal cues and fail to generalize across diverse deepfake generation techniques.
• To exploit spatiotemporal inconsistencies in local facial regions across frames—key indicators of deepfake artifacts—by reorganizing video input into patch-level representations.
• To design a flexible, plug-in spatiotemporal dropout (STD) mechanism that enhances ViT-based models through effective data augmentation and improved feature learning.
• To achieve superior detection performance across diverse deepfake datasets with strong generalization and robustness against unseen generation methods.

Proposed method

• Input videos are first extracted into facial frame sequences, then each frame is divided into non-overlapping, grid-wise facial patches to form a bag of patches.
• The bag of patches is fed into a Vision Transformer (ViT) to learn discriminative, robust representations of spatiotemporal cues across frames and patches.
• A spatiotemporal dropout (STD) operation is introduced: first, temporal dropout randomly removes frames from the sequence; then, spatial dropout randomly discards patches from the remaining frames.
• The resulting reduced bag of patches preserves full facial coverage while exposing inconsistencies in local regions across time, enhancing model sensitivity to deepfake artifacts.
• The random nature of STD generates diverse training instances per video, effectively serving as data augmentation to improve generalization and robustness.
• The STD operation is architecture-agnostic and can be seamlessly integrated into existing Vision Transformer models.

Experimental results

Research questions

• RQ1 Can modeling patch-level spatiotemporal inconsistencies across frames improve deepfake video detection beyond frame-level analysis?
• RQ2 How effective is a spatiotemporal dropout mechanism in enhancing model robustness and generalization for deepfake detection?
• RQ3 Does the proposed STD-Former framework outperform state-of-the-art methods across diverse deepfake datasets and generation techniques?
• RQ4 How do hyperparameters like temporal and spatial dropout rates affect detection performance?
• RQ5 To what extent can the STD operation be generalized across different ViT architectures?

Key findings

• The proposed spatiotemporal dropout (S+T) variant achieved the highest performance, improving accuracy and AUC over variants with only spatial or temporal dropout, demonstrating the necessity of joint spatiotemporal modeling.
• Optimal dropout rates were found to be α = 1/4 (temporal dropout rate) and β′ = 1/18 (spatial dropout rate), as performance declined when rates were too high or too low.
• On the Celeb-DF(v2) dataset, the STD-Former improved average accuracy by 11.15% and AUC by 7.76% across four ViT backbones (ViT-B16, ViT-B32, ViT-L16, ViT-L32) compared to baseline models without STD.
• The method achieved state-of-the-art performance on 25 benchmarks, demonstrating strong generalization and robustness against diverse deepfake generation methods.
• Ablation studies confirmed that the STD operation significantly enhances model robustness and generalization, with consistent improvements across different ViT architectures and network designs.
• The approach outperformed a strong baseline (TD-3DCNN) when using the same dropout variants, further validating the effectiveness of the proposed STD mechanism.