[Paper Review] FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces

Andreas Rössler, Davide Cozzolino+4 more|arXiv (Cornell University)|Mar 24, 2018

Digital Media Forensic Detection3 references362 citations

TL;DR

Introduces FaceForensics, a large-scale face-manipulation video dataset (>500k frames from 1004 videos) for forgery detection and segmentation, plus baseline benchmarks and a refinement method.

ABSTRACT

With recent advances in computer vision and graphics, it is now possible to generate videos with extremely realistic synthetic faces, even in real time. Countless applications are possible, some of which raise a legitimate alarm, calling for reliable detectors of fake videos. In fact, distinguishing between original and manipulated video can be a challenge for humans and computers alike, especially when the videos are compressed or have low resolution, as it often happens on social networks. Research on the detection of face manipulations has been seriously hampered by the lack of adequate datasets. To this end, we introduce a novel face manipulation dataset of about half a million edited images (from over 1000 videos). The manipulations have been generated with a state-of-the-art face editing approach. It exceeds all existing video manipulation datasets by at least an order of magnitude. Using our new dataset, we introduce benchmarks for classical image forensic tasks, including classification and segmentation, considering videos compressed at various quality levels. In addition, we introduce a benchmark evaluation for creating indistinguishable forgeries with known ground truth; for instance with generative refinement models.

Motivation & Objective

• Provide a large, realistic dataset of manipulated facial videos to enable data-driven forgery detection.
• Benchmark forgery classification and pixel-level segmentation under different compression regimes.
• Evaluate state-of-the-art detectors on FaceForensics and establish baselines for future work.
• Explore a supervised refinement approach to improve the realism of forged faces and assess detection robustness.

Proposed method

• Generate a dataset of over 500,000 frames from 1004 YouTube videos using the Face2Face reenactment method to create source-to-target and self-reenactment manipulations.
• Provide ground-truth per-pixel masks indicating modified regions for segmentation tasks.
• Evaluate multiple learning-based and handcrafted forgery detectors on uncompressed and compressed (easy and hard) videos.
• Adapt XceptionNet and other architectures for face-centered forgery classification and for pixel-level segmentation via a sliding-window approach.
• Propose an autoencoder-based refinement model (pre-trained on VGGFace2) to improve visual quality of forgeries and test its impact on detectability.
• Assess perceptual quality through a user study comparing raw and refined forgeries.

Experimental results

Research questions

• RQ1 How well do current state-of-the-art detection approaches perform on realistic, large-scale Face2Forensics manipulations under different compression levels?
• RQ2 Can a data-driven dataset enable robust forgery classification and segmentation for faces in videos?
• RQ3 Does a supervised autoencoder refinement improve visual quality of forgeries, and how does it affect detectability by classifiers?

Key findings

• The dataset contains over 500k frames from 1004 videos with source-to-target and self-reenactment manipulations and ground-truth masks.
• Classification performance varies by method and compression; deep models (XceptionNet) outperform handcrafted features under compression, with accuracies around 87–98% for no-c and easy-c, and up to 87.81% for hard-c.
• Forgery localization via CNNs shows strong performance on uncompressed data, but degrades with compression; XceptionNet remains most robust among tested methods.
• Self-reenactment ground-truth data enables supervised refinement training for improving fake realism, with perceptual gains shown in a user study where refined fakes are harder to detect for humans, especially under compression.
• The autoencoder-based refiner improves artifact regions (chin, nose, cheek) and illumination, but detection accuracy using refined data remains high for models trained on forged outputs.
• Quantitative results indicate that refined fakes can slightly lower detection on 128x128 inputs but still retain strong detectability by advanced detectors.