[Paper Review] The Deepfake Detection Challenge (DFDC) Preview Dataset

Brian Dolhansky, Russ Howes+3 more|arXiv (Cornell University)|Oct 19, 2019

Digital Media Forensic Detection6 references157 citations

TL;DR

The paper introduces a 5K-video DFDC preview dataset with two face-swap methods, provides a data collection process with participant consent, outlines evaluation metrics emphasizing weighted precision, and reports baseline results from TamperNet and XceptionNet models.

ABSTRACT

In this paper, we introduce a preview of the Deepfakes Detection Challenge (DFDC) dataset consisting of 5K videos featuring two facial modification algorithms. A data collection campaign has been carried out where participating actors have entered into an agreement to the use and manipulation of their likenesses in our creation of the dataset. Diversity in several axes (gender, skin-tone, age, etc.) has been considered and actors recorded videos with arbitrary backgrounds thus bringing visual variability. Finally, a set of specific metrics to evaluate the performance have been defined and two existing models for detecting deepfakes have been tested to provide a reference performance baseline. The DFDC dataset preview can be downloaded at: deepfakedetectionchallenge.ai

Motivation & Objective

• Motivate and enable research on deepfake detection with a diverse, consent-based video dataset.
• Provide a preview dataset to benchmark detection methods and establish baseline performance.
• Define evaluation metrics that account for the prevalence gap between deepfakes and real videos.
• Offer initial baseline results using simple detection models to guide future research.

Proposed method

• Describe the data collection process with crowdsourced actors ensuring diversity in gender, skin tone, and age.
• Create two face-swap methods to simulate manipulation (methods A and B) and apply filtering to swaps based on face size.
• Extract multiple 15-second clips per video and apply controlled augmentations to test clips to simulate real-world degradations.
• Define weighted precision (wP) to reflect the difference in deepfake prevalence between datasets and real-world traffic, and report log(wP) at decay levels for recall.
• Provide baseline evaluations using TamperNet, XceptionNet (Face), and XceptionNet (Full) on frame-based and video-level detection.

Experimental results

Research questions

• RQ1 What is the baseline performance of simple deepfake detection models on a diverse, consented DFDC preview dataset?
• RQ2 How do weakly supervised or frame-based vs. full-video detection approaches compare on this dataset?
• RQ3 How should evaluation metrics be adapted to reflect real-world class imbalance between fake and real videos?
• RQ4 What is the impact of standard augmentations on detected deepfakes in a controlled preview setting?

Key findings

• The DFDC Preview Dataset comprises 4,464 training clips and 780 test clips across 66 individuals with consent from participants.
• Two swap methods (A and B) are used to generate face swaps, with filtering based on face size ratio to avoid near-identity swaps in close-up frames.
• Video augmentations (lower FPS to 15, reduce resolution to 1/4, and lower encoding quality) are applied to test clips to simulate real-world degradation.
• Baseline results show TamperNet achieving 0.833 precision and 0.033 recall with log(WP) = -3.044 on the dataset when optimized for log(WP).
• XceptionNet (Face) achieves 0.930 precision and 0.084 recall with log(WP) = -2.140 on the same setup, while XceptionNet (Full) achieves 0.784 precision and 0.268 recall with log(WP) = -3.352.
• The paper defines a weighted precision metric (log(WP)) to reflect realistic deepfake prevalence differences between datasets and organic traffic.