[Paper Review] ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis

Yinan He, Bei Gan+7 more|arXiv (Cornell University)|Mar 9, 2021

Digital Media Forensic Detection57 references4 citations

TL;DR

ForgeryNet is a large-scale, multi-task benchmark dataset comprising 2.9 million images and 221,247 videos for comprehensive face forgery analysis. It supports four tasks—image/video forgery classification, spatial forgery localization, and temporal forgery localization—using 15 image-level and 8 video-level manipulation methods with 36 re-rendering perturbations, enabling robust evaluation of forgery detection models across diverse, real-world conditions.

ABSTRACT

The rapid progress of photorealistic synthesis techniques has reached at a critical point where the boundary between real and manipulated images starts to blur. Thus, benchmarking and advancing digital forgery analysis have become a pressing issue. However, existing face forgery datasets either have limited diversity or only support coarse-grained analysis. To counter this emerging threat, we construct the ForgeryNet dataset, an extremely large face forgery dataset with unified annotations in image- and video-level data across four tasks: 1) Image Forgery Classification, including two-way (real / fake), three-way (real / fake with identity-replaced forgery approaches / fake with identity-remained forgery approaches), and n-way (real and 15 respective forgery approaches) classification. 2) Spatial Forgery Localization, which segments the manipulated area of fake images compared to their corresponding source real images. 3) Video Forgery Classification, which re-defines the video-level forgery classification with manipulated frames in random positions. This task is important because attackers in real world are free to manipulate any target frame. and 4) Temporal Forgery Localization, to localize the temporal segments which are manipulated. ForgeryNet is by far the largest publicly available deep face forgery dataset in terms of data-scale (2.9 million images, 221,247 videos), manipulations (7 image-level approaches, 8 video-level approaches), perturbations (36 independent and more mixed perturbations) and annotations (6.3 million classification labels, 2.9 million manipulated area annotations and 221,247 temporal forgery segment labels). We perform extensive benchmarking and studies of existing face forensics methods and obtain several valuable observations.

Motivation & Objective

• Address the growing threat of photorealistic face forgery by creating a scalable, diverse, and realistic benchmark for detection research.
• Overcome limitations of existing datasets, which suffer from small scale, limited diversity in forgery methods, and lack of realistic perturbations.
• Enable comprehensive evaluation of forgery detection models across image-level and video-level tasks, including spatial and temporal localization.
• Support both binary and fine-grained classification (e.g., n-way with 15 forgery types) to reflect real-world detection demands.
• Provide rich annotations across multiple levels—image, video, and temporal segments—to support end-to-end forgery analysis pipelines.

Proposed method

• Construct ForgeryNet using wild, diverse original data from four public datasets with Creative Commons licensing, covering varied identities, expressions, lighting, and angles.
• Apply 15 distinct deep learning-based image forgery methods (e.g., face swap, reenactment, editing) and 8 video-level methods using models like GANs, RNNs, and autoencoders.
• Apply 36 independent and mixed re-rendering perturbations (e.g., compression, blur, noise) to simulate real-world media transmission and degradation.
• Annotate data with 6.3M classification labels, 2.9M spatial segmentation masks for manipulated regions, and 221,247 temporal forgery segment labels.
• Design four evaluation tasks: (1) two-way, three-way, and n-way image forgery classification; (2) spatial forgery localization; (3) video-level forgery classification with random frame manipulation; (4) temporal forgery localization.
• Train and evaluate state-of-the-art models (e.g., SlowFast, X3D-M, BSN, BMN) on the dataset using standard protocols with cross-entropy loss, end-to-end training, and multi-crop inference strategies.

Experimental results

Research questions

• RQ1 How does model performance vary across different levels of data perturbation and re-rendering in real-world forgery detection?
• RQ2 To what extent do existing forgery detection models generalize to unseen forgery methods and complex, mixed perturbations?
• RQ3 How effective are models at localizing manipulated regions in images and temporal segments in videos, especially when manipulation is non-uniformly distributed?
• RQ4 What is the impact of temporal continuity and frame ordering on video-based forgery classification?
• RQ5 Can the proposed benchmark reveal limitations in current models that are not exposed by smaller, less diverse datasets?

Key findings

• ForgeryNet is the largest publicly available deep face forgery dataset, with 2.9 million images and 221,247 videos, supporting 15 image-level and 8 video-level forgery methods.
• The dataset includes 36 distinct re-rendering perturbations, simulating real-world media degradation and increasing detection difficulty.
• Video-level forgery classification models show robustness to temporal shuffling, with AUC scores of 94.11 under weak shuffling (every 16 frames), indicating reliance on non-temporal cues.
• Image-level models are more sensitive to data augmentation than video-level models, with accuracy dropping from 84.39% (weak) to 85.04% (enhanced) under strong augmentation.
• Boundary-aware temporal localization models (e.g., BMN) achieve high precision in identifying transition points between real and fake segments, with SoftNMS suppressing redundant predictions.
• The benchmark reveals that state-of-the-art models still struggle with fine-grained forgery type classification and generalization across unseen manipulation techniques and perturbations.