[Paper Review] On Improving Cross-dataset Generalization of Deepfake Detectors

Aakash Varma Nadimpalli, Ajita Rattani|arXiv (Cornell University)|Apr 8, 2022

Digital Media Forensic Detection4 citations

TL;DR

This paper proposes a hybrid supervised-reinforcement learning framework that applies image-specific test-time data augmentations to improve cross-dataset generalization in deepfake detection. By using a reinforcement learning agent to select the top-k augmentations per test image and averaging their CNN-based classification scores, the method achieves state-of-the-art AUC of 0.669 on Celeb-DF, significantly outperforming baseline CNNs and existing methods in cross-dataset evaluation.

ABSTRACT

Facial manipulation by deep fake has caused major security risks and raised severe societal concerns. As a countermeasure, a number of deep fake detection methods have been proposed recently. Most of them model deep fake detection as a binary classification problem using a backbone convolutional neural network (CNN) architecture pretrained for the task. These CNN-based methods have demonstrated very high efficacy in deep fake detection with the Area under the Curve (AUC) as high as 0.99. However, the performance of these methods degrades significantly when evaluated across datasets. In this paper, we formulate deep fake detection as a hybrid combination of supervised and reinforcement learning (RL) to improve its cross-dataset generalization performance. The proposed method chooses the top-k augmentations for each test sample by an RL agent in an image-specific manner. The classification scores, obtained using CNN, of all the augmentations of each test image are averaged together for final real or fake classification. Through extensive experimental validation, we demonstrate the superiority of our method over existing published research in cross-dataset generalization of deep fake detectors, thus obtaining state-of-the-art performance.

Motivation & Objective

• To address the significant performance drop of CNN-based deepfake detectors in cross-dataset evaluation due to domain shift.
• To improve the robustness and transferability of deepfake detection models across diverse datasets with varying image quality and generation techniques.
• To reduce the impact of domain shift by learning adaptive, image-specific data augmentations at test time.
• To achieve state-of-the-art performance in cross-dataset deepfake detection generalization.

Proposed method

• A reinforcement learning (RL) agent is trained to select the top-k data augmentations for each test image based on a learned policy.
• The RL agent uses a Q-table or policy network (e.g., PPO) to choose augmentations that maximize expected reward, defined by improved detection confidence.
• For each test image, multiple augmented versions are generated, and their CNN-based classification scores are averaged for final prediction.
• The method is applied post-training, making it compatible with any pre-trained CNN backbone (e.g., EfficientNet V2-L, XceptionNet).
• The RL agent is trained on a source dataset (e.g., FaceForensics++) and then applied to target datasets (e.g., Celeb-DF, DeeperForensics-1.0) without further fine-tuning.
• The approach enables image-specific augmentation selection, adapting to the visual quality and forgery characteristics of each input.

Experimental results

Research questions

• RQ1 Can test-time data augmentation guided by reinforcement learning improve cross-dataset generalization of deepfake detectors?
• RQ2 Does image-specific augmentation selection outperform random or fixed augmentation strategies in reducing domain shift effects?
• RQ3 What is the optimal number of augmentations (k) for maximizing detection performance across datasets?
• RQ4 Can a policy learned on one dataset generalize effectively to unseen datasets with different deepfake generation techniques?

Key findings

• The proposed method achieved an AUC of 0.669 on Celeb-DF, a 0.018 improvement over baseline CNNs, demonstrating state-of-the-art performance in cross-dataset evaluation.
• The top-3 augmentations yielded the best performance, with AUC of 0.994 on FaceForensics++ and 0.669 on Celeb-DF, indicating optimal k=3.
• The method reduced EER to 0.362 on Celeb-DF, a 0.021 decrease from baseline CNNs, showing improved robustness.
• The ablation study confirmed that systematic RL-based augmentation selection outperformed random or fixed augmentation strategies.
• The policy learned on FaceForensics++ generalized effectively to Celeb-DF and DeeperForensics-1.0, confirming transferability across datasets.
• Even with a high intra-dataset AUC of 0.994, the baseline CNNs dropped to 0.482 on Celeb-DF, highlighting the critical need for cross-dataset generalization.