[Paper Review] The DeepFake Detection Challenge (DFDC) Dataset

Brian Dolhansky|arXiv (Cornell University)|Jun 12, 2020

Generative Adversarial Networks and Image Synthesis24 references50 citations

TL;DR

The paper introduces the DFDC dataset and Kaggle competition, detailing dataset scale, construction methods, augmentations, and baseline results showing generalization to real Deepfakes.

ABSTRACT

Deepfakes are a recent off-the-shelf manipulation technique that allows anyone to swap two identities in a single video. In addition to Deepfakes, a variety of GAN-based face swapping methods have also been published with accompanying code. To counter this emerging threat, we have constructed an extremely large face swap video dataset to enable the training of detection models, and organized the accompanying DeepFake Detection Challenge (DFDC) Kaggle competition. Importantly, all recorded subjects agreed to participate in and have their likenesses modified during the construction of the face-swapped dataset. The DFDC dataset is by far the largest currently and publicly available face swap video dataset, with over 100,000 total clips sourced from 3,426 paid actors, produced with several Deepfake, GAN-based, and non-learned methods. In addition to describing the methods used to construct the dataset, we provide a detailed analysis of the top submissions from the Kaggle contest. We show although Deepfake detection is extremely difficult and still an unsolved problem, a Deepfake detection model trained only on the DFDC can generalize to real "in-the-wild" Deepfake videos, and such a model can be a valuable analysis tool when analyzing potentially Deepfaked videos. Training, validation and testing corpuses can be downloaded from https://ai.facebook.com/datasets/dfdc.

Motivation & Objective

• Motivate the need for scalable Deepfake detection in the face of rising public-facing manipulation tools.
• Create a large, consented DFDC dataset with diverse methods and real-world variations.
• Publicly benchmark detection models to assess generalization to real-world Deepfakes.
• Provide insights into augmentation, distractors, and evaluation metrics suitable for imbalanced Deepfake detection.

Proposed method

• Assembled a large-scale, consent-based face-swap video dataset with 3,426 paid actors (≈48,190 videos, 25 TB raw data).
• Employed multiple face-swapping methods (DFAE, MM/NN, NTH, FSGAN, StyleGAN) and post-processing to generate Deepfakes.
• Applied extensive data augmentation and distractors (geometric/color transforms, overlays, Dog/Flower filters) to 70% of videos.
• Implemented a two-test-set regime (public test for leaderboard, private test for final evaluation) and a weighted precision metric alongside log loss.
• Hosted a large Kaggle competition with 2,114 teams and a private test distribution to measure real-world performance.
• Compared various architectures and reported top-5 results on precision at multiple recall levels.

Experimental results

Research questions

• RQ1 How well do detectors trained on the DFDC dataset generalize to real, unseen Deepfake videos?
• RQ2 What is the impact of different face-swapping methods on detectability and temporal coherence?
• RQ3 How do augmentations and distractors affect detection performance under realistic class imbalance?
• RQ4 What metrics best reflect detector utility in real-world deployment (precision under skewed distributions)?

Key findings

• Training on DFDC yields generalization to real videos, with average precision 0.753 and ROC-AUC 0.734 on real videos.
• Public leaderboard performance correlates with private test performance, indicating generalization trends.
• Top-5 submissions achieved strong DFDC performance with log losses around 0.43 on DFDC videos; real-video performance is lower but still indicative of generalization.
• DFAE-based methods tended to produce higher-quality swaps across varied conditions compared to GAN-like methods, which struggled in uncontrolled settings.
• StyleGAN-based swaps exhibited inconsistent gaze and illumination matching, performing worst among evaluated methods.