[Paper Review] UCF: Uncovering Common Features for Generalizable Deepfake Detection

Zhiyuan Yan, Yong Zhang+2 more|arXiv (Cornell University)|Apr 27, 2023

Digital Media Forensic Detection4 citations

TL;DR

This paper proposes a disentanglement framework, UCF, that uncovers common forgery features across diverse deepfake methods to improve generalization. By using multi-task learning, a conditional decoder, and contrastive regularization, the model isolates method-agnostic features, achieving state-of-the-art performance on unseen datasets with up to 94.5% AUC on DFD and 86.6% average AUC across benchmarks.

ABSTRACT

Deepfake detection remains a challenging task due to the difficulty of generalizing to new types of forgeries. This problem primarily stems from the overfitting of existing detection methods to forgery-irrelevant features and method-specific patterns. The latter has been rarely studied and not well addressed by previous works. This paper presents a novel approach to address the two types of overfitting issues by uncovering common forgery features. Specifically, we first propose a disentanglement framework that decomposes image information into three distinct components: forgery-irrelevant, method-specific forgery, and common forgery features. To ensure the decoupling of method-specific and common forgery features, a multi-task learning strategy is employed, including a multi-class classification that predicts the category of the forgery method and a binary classification that distinguishes the real from the fake. Additionally, a conditional decoder is designed to utilize forgery features as a condition along with forgery-irrelevant features to generate reconstructed images. Furthermore, a contrastive regularization technique is proposed to encourage the disentanglement of the common and specific forgery features. Ultimately, we only utilize the common forgery features for the purpose of generalizable deepfake detection. Extensive evaluations demonstrate that our framework can perform superior generalization than current state-of-the-art methods.

Motivation & Objective

• To address the generalization problem in deepfake detection caused by overfitting to forgery-irrelevant and method-specific features.
• To disentangle image features into content, method-specific forgery, and common forgery components for robust detection.
• To develop a plug-and-play framework that enhances generalization across diverse backbones and unseen forgery types.

Proposed method

• A multi-task learning framework that jointly performs binary real/fake classification and multi-class forgery method classification to disentangle content from forgery features.
• A conditional decoder that reconstructs images using both content and forgery features as conditions, ensuring faithful disentanglement.
• A contrastive regularization loss that explicitly encourages separation between method-specific and common forgery features.
• The final detector uses only the common forgery features, discarding content and method-specific patterns to prevent overfitting.
• The framework is compatible with various backbones, including Xception, ConvNeXt, ResNet, and EfficientNet, demonstrating plug-and-play capability.
• t-SNE visualization confirms that the common features capture shared forgery patterns across methods, while content features remain invariant to real/fake labels.

Experimental results

Research questions

• RQ1 Can a disentanglement framework effectively separate forgery-irrelevant content from method-specific and common forgery features?
• RQ2 Does isolating only common forgery features lead to superior generalization on unseen deepfake datasets?
• RQ3 How does the proposed contrastive regularization improve disentanglement between specific and common forgery patterns?
• RQ4 Can the framework maintain high performance across diverse backbone architectures?
• RQ5 To what extent does the conditional decoder enhance feature disentanglement and reconstruction fidelity?

Key findings

• The UCF framework achieves 85.2% average AUC across datasets, significantly outperforming the baseline Xception model (68.3% average AUC).
• On the DFD dataset, the method reaches 94.5% AUC, demonstrating strong generalization to unseen forgeries.
• With the ConvNeXt backbone, the average AUC improves to 86.6%, confirming the framework's effectiveness across different architectures.
• t-SNE visualization confirms that the common forgery features capture shared patterns across different forgery methods, while content features remain invariant to real/fake labels.
• The ablation study shows that using only common features yields better performance than using all forgery features, proving that method-specific patterns degrade generalization.
• The framework is plug-and-play, as it enhances detection performance on ResNet and EfficientNet backbones without architectural modifications.