[Paper Review] Recurrent Convolutional Strategies for Face Manipulation Detection in Videos

Ekraam Sabir, Jiaxin Cheng+4 more|arXiv (Cornell University)|May 2, 2019

Digital Media Forensic Detection47 references337 citations

TL;DR

The paper proposes a recurrent-convolutional framework with face alignment to detect manipulated faces in videos, achieving state-of-the-art accuracy on FaceForensics++ by leveraging temporal information.

ABSTRACT

The spread of misinformation through synthetically generated yet realistic images and videos has become a significant problem, calling for robust manipulation detection methods. Despite the predominant effort of detecting face manipulation in still images, less attention has been paid to the identification of tampered faces in videos by taking advantage of the temporal information present in the stream. Recurrent convolutional models are a class of deep learning models which have proven effective at exploiting the temporal information from image streams across domains. We thereby distill the best strategy for combining variations in these models along with domain specific face preprocessing techniques through extensive experimentation to obtain state-of-the-art performance on publicly available video-based facial manipulation benchmarks. Specifically, we attempt to detect Deepfake, Face2Face and FaceSwap tampered faces in video streams. Evaluation is performed on the recently introduced FaceForensics++ dataset, improving the previous state-of-the-art by up to 4.55% in accuracy.

Motivation & Objective

• Motivate detection of manipulated faces in videos by exploiting temporal coherence in addition to spatial cues.
• Evaluate how face preprocessing (alignment) impacts detection accuracy.
• Explore architectural choices (backbone CNNs and recurrent design) to maximize detection performance on video manipulation benchmarks.

Proposed method

• Crop and align face regions from video frames using landmark-based alignment or Spatial Transformer Networks (STN).
• Construct a recurrent-convolutional detector that operates on face tubes (sequences of aligned crops).
• Experiment with backbone CNNs (DenseNet and ResNet variants) followed by GRU-based recurrence.
• Compare single-recurrence vs multi-level recurrence to capture micro, meso, and macro features.
• Train end-to-end on FF++ with binary real/fake supervision; use Adam optimizer with 1e-4 learning rate.

Experimental results

Research questions

• RQ1 Can temporal information in videos improve face manipulation detection beyond frame-level cues?
• RQ2 Does explicit landmark-based alignment outperform implicit alignment (STN) for this task?
• RQ3 Which backbone (DenseNet vs ResNet) and temporal strategy (single vs multi-recurrence; bidirectional vs unidirectional) yield the best performance across manipulation types?
• RQ4 Is multi-level recurrence beneficial or does it risk overfitting given FF++ data size?

Key findings

• DenseNet with landmark-based alignment and bidirectional GRU recurrence achieves best performance.
• Face alignment improves detection accuracy over no-alignment baselines.
• Using a sequence of frames (five-frame input) outperforms single-frame input.
• Bidirectional recurrence outperforms uni-directional recurrence.
• STN-based alignment and multi-recurrence strategies did not improve performance and can hurt stability or lead to overfitting.