[Paper Review] Generalized Face Forgery Detection via Adaptive Learning for Pre-trained Vision Transformer

Anwei Luo, Rizhao Cai+5 more|arXiv (Cornell University)|Sep 20, 2023

Face recognition and analysisComputer Science3 citations

TL;DR

This paper proposes Forgery-aware Adaptive Vision Transformer (FA-ViT), a parameter-efficient method for face forgery detection that preserves pre-trained Vision Transformer (ViT) knowledge while adapting to forgery-specific features via two modules: Local-aware Forgery Injector (LFI) with Neighborhood-Preserving Cross Attention (NPCA) for local artifact detection, and Global-aware Forgery Adaptor (GFA) for global domain adaptation. FA-ViT achieves state-of-the-art performance in cross-dataset and cross-manipulation generalization, with 99.60% AUC on FaceForensics++ and 93.83% on Celeb-DF.

ABSTRACT

With the rapid progress of generative models, the current challenge in face forgery detection is how to effectively detect realistic manipulated faces from different unseen domains. Though previous studies show that pre-trained Vision Transformer (ViT) based models can achieve some promising results after fully fine-tuning on the Deepfake dataset, their generalization performances are still unsatisfactory. One possible reason is that fully fine-tuned ViT-based models may disrupt the pre-trained features [1, 2] and overfit to some data-specific patterns [3]. To alleviate this issue, we present a extbf{F}orgery-aware extbf{A}daptive extbf{Vi}sion extbf{T}ransformer (FA-ViT) under the adaptive learning paradigm, where the parameters in the pre-trained ViT are kept fixed while the designed adaptive modules are optimized to capture forgery features. Specifically, a global adaptive module is designed to model long-range interactions among input tokens, which takes advantage of self-attention mechanism to mine global forgery clues. To further explore essential local forgery clues, a local adaptive module is proposed to expose local inconsistencies by enhancing the local contextual association. In addition, we introduce a fine-grained adaptive learning module that emphasizes the common compact representation of genuine faces through relationship learning in fine-grained pairs, driving these proposed adaptive modules to be aware of fine-grained forgery-aware information. Extensive experiments demonstrate that our FA-ViT achieves state-of-the-arts results in the cross-dataset evaluation, and enhances the robustness against unseen perturbations. Particularly, FA-ViT achieves 93.83\% and 78.32\% AUC scores on Celeb-DF and DFDC datasets in the cross-dataset evaluation. The code and trained model have been released at: https://github.com/LoveSiameseCat/FAViT.

Motivation & Objective

• Address the poor generalization of ViT-based face forgery detectors in cross-dataset evaluations due to catastrophic forgetting and overfitting on limited Deepfake data.
• Preserve pre-trained ViT knowledge by freezing its parameters while injecting forgery-specific knowledge through parameter-efficient adaptation.
• Improve detection robustness against unseen manipulation types and challenging conditions (e.g., large poses, poor lighting) through adaptive feature learning.
• Enhance fine-grained feature learning by compacting genuine face feature distributions using a novel Single Domain Pairwise Learning (SDPL) strategy.
• Bridge the domain gap between ImageNet-level classification and fine-grained forgery detection through global and local forgery-aware adaptation.

Proposed method

• Freeze all pre-trained ViT parameters to retain general visual representations and prevent catastrophic forgetting.
• Introduce the Local-aware Forgery Injector (LFI) to capture local forgery artifacts using convolutional inductive bias and inject them into ViT via Neighborhood-Preserving Cross Attention (NPCA).
• Design the Global-aware Forgery Adaptor (GFA) to learn adaptive transformations in the Multi-Head Self-Attention (MHSA) layer, aligning pre-trained features with forgery-relevant representations.
• Implement identity mapping initialization in LFI and GFA to stabilize training by ensuring initial outputs are near zero.
• Propose Single Domain Pairwise Learning (SDPL) that constructs fine-grained real face pairs sharing background and posture but differing in identity, with the real class classifier vector used as anchor to refine feature compactness.
• Use Grad-CAM++ to visualize attention maps and validate that FA-ViT focuses on actual manipulation regions (e.g., mouth, eyes) rather than central facial regions.

Experimental results

Research questions

• RQ1 Can we improve the generalization of ViT-based face forgery detectors across unseen datasets and manipulation types without fine-tuning the entire model?
• RQ2 How can we effectively inject local forgery cues into a pre-trained ViT while preserving its general representation capacity?
• RQ3 To what extent can global adaptive adaptation in self-attention layers bridge the domain gap between ImageNet classification and forgery detection?
• RQ4 Does a novel pairwise learning strategy that emphasizes fine-grained real face similarity improve feature compactness and detection robustness?
• RQ5 Can the proposed FA-ViT maintain high attention on manipulation regions under challenging conditions such as large poses or poor lighting?

Key findings

• FA-ViT achieves 99.60% AUC on FaceForensics++ (FF++), outperforming the next best variant by 0.06% and demonstrating state-of-the-art performance in cross-dataset evaluation.
• On the Celeb-DF (CDF) dataset, FA-ViT achieves 93.83% AUC, a significant improvement over the variant without identity mapping initialization (91.38%) and the baseline ViT.
• On the Wild-Deepfake (WDF) dataset, FA-ViT achieves 84.32% AUC, showing strong generalization to unseen manipulation methods.
• t-SNE visualization confirms that FA-ViT maintains tight clustering of genuine faces across different datasets, indicating robust and generalized feature learning.
• Grad-CAM++ visualizations show that FA-ViT consistently focuses on manipulation-specific regions (e.g., mouth for Face2Face, eyes for Deepfakes), while ViT fails to localize such artifacts.
• Ablation studies confirm that identity mapping initialization in both LFI and GFA is critical—removing it degrades performance by up to 2.5% on CDF, indicating its role in training stability and feature consistency.