[Paper Review] A New Approach to Improve Learning-based Deepfake Detection in Realistic Conditions

Yuhang Lu, Touradj Ebrahimi|arXiv (Cornell University)|Mar 22, 2022

Digital Media Forensic Detection4 citations

TL;DR

This paper proposes a novel data augmentation scheme based on real-world image degradation processes—such as JPEG compression, noise, blur, and gamma correction—to improve the robustness of learning-based deepfake detectors. By training models with this stochastic, cascaded augmentation, the method significantly enhances generalization to unseen distortions and processing operations, achieving higher AUC scores on corrupted and unseen datasets while maintaining strong performance on original benchmarks.

ABSTRACT

Deep convolutional neural networks have achieved exceptional results on multiple detection and recognition tasks. However, the performance of such detectors are often evaluated in public benchmarks under constrained and non-realistic situations. The impact of conventional distortions and processing operations found in imaging workflows such as compression, noise, and enhancement are not sufficiently studied. Currently, only a few researches have been done to improve the detector robustness to unseen perturbations. This paper proposes a more effective data augmentation scheme based on real-world image degradation process. This novel technique is deployed for deepfake detection tasks and has been evaluated by a more realistic assessment framework. Extensive experiments show that the proposed data augmentation scheme improves generalization ability to unpredictable data distortions and unseen datasets.

Motivation & Objective

• To address the lack of robustness in deepfake detectors when faced with real-world image distortions and processing operations such as compression, noise, and blur.
• To develop a data augmentation scheme that mimics natural image degradation processes to improve detector generalization under realistic conditions.
• To introduce a rigorous assessment framework for evaluating detector robustness beyond standard benchmarks, incorporating diverse processing operations and corruption levels.
• To evaluate the effectiveness of the proposed augmentation in improving performance on unseen datasets and under severe distortions.
• To demonstrate that stochastic, cascaded augmentation preserves performance on clean data while significantly boosting resilience to real-world corruptions.

Proposed method

• The proposed data augmentation scheme applies a sequence of realistic image processing operations—including JPEG compression, Gaussian noise, Gaussian blur, and gamma correction—in a stochastic, cascaded manner during training.
• The augmentation is applied in a randomized order and with varying severity levels to simulate diverse real-world imaging workflows.
• The method is integrated into the training pipeline without modifying the model architecture, making it compatible with existing deepfake detection frameworks.
• A new assessment framework is designed to evaluate detectors on test data subjected to controlled, realistic distortions and processing operations, isolated from training data.
• The framework evaluates performance using standard metrics like AUC, accuracy, and F1-score across multiple corruption types and severity levels.
• Ablation studies compare the proposed scheme against variants, including single-operation augmentation and non-stochastic implementations, to isolate the impact of stochasticity and cascading.

Experimental results

Research questions

• RQ1 How does training with a realistic, cascaded data augmentation scheme affect deepfake detector performance under various real-world distortions?
• RQ2 To what extent does the proposed augmentation improve generalization to unseen datasets and unseen processing operations?
• RQ3 How does the stochastic implementation of the augmentation scheme compare to deterministic or single-operation alternatives in terms of robustness and performance on clean data?
• RQ4 What is the impact of combining multiple realistic image processing operations on detector robustness?
• RQ5 Can the proposed assessment framework effectively measure and quantify detector robustness in realistic imaging conditions?

Key findings

• The proposed data augmentation scheme significantly improves detector robustness across all tested distortions, including JPEG compression, Gaussian noise, blur, and gamma correction, with consistent AUC improvements on corrupted data.
• On the Celeb-DFv2 dataset, the Capsule-Forensics model trained with the proposed augmentation achieved an AUC of 71.86%, a 17.47% improvement over the baseline (54.39%) trained on raw data.
• The XceptionNet detector showed a 23.88% AUC improvement on Celeb-DFv2 when trained with the proposed augmentation (73.88%) compared to the baseline (50.00%) on raw data.
• The ablation study confirmed that the stochastic, cascaded nature of the augmentation is critical: non-stochastic training led to a performance drop on clean data, while single-operation augmentation (e.g., noise only) improved robustness only for that specific corruption.
• Even under high-severity distortions, models trained with the proposed scheme maintained significantly higher AUC scores than baseline models, demonstrating strong generalization to unpredictable corruptions.
• The assessment framework successfully revealed performance degradation in standard detectors under realistic conditions, highlighting the limitations of conventional benchmarking.