[Paper Review] Towards General Visual-Linguistic Face Forgery Detection

Ke Sun, Chen Shen+5 more|arXiv (Cornell University)|Jul 31, 2023

Face recognition and analysis4 citations

TL;DR

This paper proposes Visual-Linguistic Face Forgery Detection (VLFFD), a novel multimodal framework that uses fine-grained sentence-level prompts as supervisory signals to improve generalization and interpretability in deepfake detection. By generating mixed forgery images with semantic annotations via the Prompt Forgery Image Generator (PFIG) and training with a Coarse-and-Fine Co-training (C2F) framework, VLFFD achieves a 13% AUC improvement over baselines on unseen test data and enables interpretable, reasoning-based detection when integrated with multimodal LLMs.

ABSTRACT

Deepfakes are realistic face manipulations that can pose serious threats to security, privacy, and trust. Existing methods mostly treat this task as binary classification, which uses digital labels or mask signals to train the detection model. We argue that such supervisions lack semantic information and interpretability. To address this issues, in this paper, we propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation. Since text annotations are not available in current deepfakes datasets, VLFFD first generates the mixed forgery image with corresponding fine-grained prompts via Prompt Forgery Image Generator (PFIG). Then, the fine-grained mixed data and coarse-grained original data and is jointly trained with the Coarse-and-Fine Co-training framework (C2F), enabling the model to gain more generalization and interpretability. The experiments show the proposed method improves the existing detection models on several challenging benchmarks. Furthermore, we have integrated our method with multimodal large models, achieving noteworthy results that demonstrate the potential of our approach. This integration not only enhances the performance of our VLFFD paradigm but also underscores the versatility and adaptability of our method when combined with advanced multimodal technologies, highlighting its potential in tackling the evolving challenges of deepfake detection.

Motivation & Objective

• To address the lack of semantic supervision and interpretability in existing binary-label face forgery detection methods.
• To improve model generalization by leveraging fine-grained textual annotations for unseen forgery types.
• To enable interpretable detection by generating explainable, sentence-level justifications for forgery predictions.
• To explore the integration of visual-linguistic paradigms with multimodal large language models (LLMs) for enhanced detection performance.
• To create a scalable pipeline for generating image-text pairs with fine-grained forgery annotations from existing datasets.

Proposed method

• Proposes the Prompt Forgery Image Generator (PFIG) to automatically generate mixed forgery images by disentangling and recombining real and fake face pairs based on quantitative criteria.
• Designs fine-grained, sentence-level prompts describing forgery attributes (e.g., unnatural eye shape, inconsistent lighting) to serve as semantic supervision.
• Introduces the Coarse-and-Fine Co-training (C2F) framework that jointly trains on both coarse-grained original data and fine-grained mixed data in a contrastive, multimodal learning setting.
• Utilizes CLIP as the backbone for visual-linguistic feature alignment, enabling zero-shot transfer and improved robustness to unseen forgery types.
• Fine-tunes MiniGPT-4 using PFIG-generated image-text pairs with reasoning-based prompts to enable LLMs to provide interpretable justifications for detection decisions.
• Employs a dual-training strategy: one for end-to-end detection and another for LLM fine-tuning, both using the same semantic annotations for consistency.

Experimental results

Research questions

• RQ1 Can fine-grained, sentence-level textual prompts improve the generalization of face forgery detection models beyond binary labels?
• RQ2 How does multimodal learning with visual-linguistic supervision enhance interpretability in forgery detection?
• RQ3 Can the proposed PFIG module generate realistic, semantically meaningful image-text pairs for deepfake data without human annotation?
• RQ4 To what extent does integrating VLFFD with multimodal LLMs improve detection performance and reasoning capability?
• RQ5 Does the C2F co-training framework effectively leverage both coarse and fine-grained data to boost robustness on unseen forgery types?

Key findings

• VLFFD achieves a 13% improvement in AUC over state-of-the-art baselines on unseen test data, demonstrating superior generalization to unseen forgery types.
• The model outputs not only a real/fake prediction but also identifies specific forgery regions and attributes through interpretable sentence-level explanations.
• Fine-tuning MiniGPT-4 with PFIG-generated data enables the LLM to provide accurate, detailed justifications for its detection decisions, unlike the original model or those fine-tuned on coarse labels.
• The C2F co-training framework effectively combines coarse and fine-grained data, leading to more robust feature learning and improved alignment between visual and linguistic signals.
• The PFIG module successfully generates semantically meaningful, diverse image-text pairs from existing datasets, enabling scalable data augmentation for multimodal training.
• Integration with multimodal LLMs shows significant promise, with the fine-tuned model outperforming both the original LLM and models trained on coarse labels in both accuracy and reasoning quality.