[Paper Review] Multimodal Graph Learning for Deepfake Detection

Zhiyuan Yan, Peng Sun+5 more|arXiv (Cornell University)|Sep 12, 2022

Emotion and Mood Recognition4 citations

TL;DR

This paper proposes Multimodal Graph Learning (MGL), a novel deepfake detection framework that integrates spatial, frequency, landmark, and temporal features via graph neural networks (GNNs) and adaptive fusion modules. By modeling facial landmarks and frame-level inconsistencies as graphs and using a bi-directional cross-modal transformer with adaptive gating, MGL achieves state-of-the-art generalization and robustness on FF++, CelebDF, and DFD benchmarks, outperforming existing methods in detecting unseen manipulation types and resisting external perturbations.

ABSTRACT

Existing deepfake detectors face several challenges in achieving robustness and generalization. One of the primary reasons is their limited ability to extract relevant information from forgery videos, especially in the presence of various artifacts such as spatial, frequency, temporal, and landmark mismatches. Current detectors rely on pixel-level features that are easily affected by unknown disturbances or facial landmarks that do not provide sufficient information. Furthermore, most detectors cannot utilize information from multiple domains for detection, leading to limited effectiveness in identifying deepfake videos. To address these limitations, we propose a novel framework, namely Multimodal Graph Learning (MGL) that leverages information from multiple modalities using two GNNs and several multimodal fusion modules. At the frame level, we employ a bi-directional cross-modal transformer and an adaptive gating mechanism to combine the features from the spatial and frequency domains with the geometric-enhanced landmark features captured by a GNN. At the video level, we use a Graph Attention Network (GAT) to represent each frame in a video as a node in a graph and encode temporal information into the edges of the graph to extract temporal inconsistency between frames. Our proposed method aims to effectively identify and utilize distinguishing features for deepfake detection. We evaluate the effectiveness of our method through extensive experiments on widely-used benchmarks and demonstrate that our method outperforms the state-of-the-art detectors in terms of generalization ability and robustness against unknown disturbances.

Motivation & Objective

• To address the limited generalization and robustness of existing deepfake detectors that rely on single-modality, pixel-level features.
• To overcome overfitting to method-specific artifacts and forgery-irrelevant textures by leveraging multi-modal, graph-based representations.
• To improve detection performance by modeling geometric relationships in facial landmarks and temporal inconsistencies between frames using graph neural networks.
• To develop adaptive fusion mechanisms that dynamically weigh spatial and frequency features based on content relevance.
• To demonstrate the effectiveness of graph-based modeling for both local (frame-level) and global (video-level) forgery pattern recognition.

Proposed method

• The framework employs a Spatial-Frequency Fusion (SFF) module using a bi-directional cross-modal transformer and adaptive gating to combine spatial and frequency domain features at the frame level.
• A Landmark Graph Learning (LGL) module uses a Graph Attention Network (GAT) to encode facial landmarks as a graph, capturing geometric dependencies between key points.
• A Temporal Graph Learning (TGL) module constructs a video-level graph where each frame is a node and edges encode temporal inconsistencies through learned attention weights.
• The model integrates features from the FM (frequency module), LGL, TGL, and SFF modules into a unified representation for final classification.
• The framework uses two GNNs—one for landmark geometry and one for temporal dynamics—enabling joint modeling of local and global forgery patterns.
• Adaptive gating and cross-attention mechanisms allow dynamic feature weighting, improving robustness to varying artifact distributions.

Experimental results

Research questions

• RQ1 Can graph-based modeling of facial landmarks improve deepfake detection beyond simple coordinate sequences?
• RQ2 Can temporal inconsistencies between frames be effectively captured using a graph-based approach with learned edge representations?
• RQ3 Does adaptive fusion of spatial and frequency features outperform fixed or unidirectional fusion strategies in deepfake detection?
• RQ4 Can a multimodal graph learning framework enhance generalization across diverse deepfake manipulation methods and unseen datasets?
• RQ5 To what extent does incorporating multiple modalities (spatial, frequency, landmark, temporal) improve robustness against external perturbations like lighting and noise?

Key findings

• The proposed MGL framework achieves state-of-the-art performance on the FF++, CelebDF, and DFD benchmarks, demonstrating superior generalization across unseen manipulation methods.
• The SFF module with bi-directional attention and adaptive gating outperforms both linear addition and M2TR fusion, showing improved detection accuracy through dynamic feature weighting.
• The LGL module significantly enhances performance by modeling geometric relationships between facial landmarks, outperforming baseline coordinate-based landmark encoding.
• The TGL module effectively captures temporal inconsistencies between frames, contributing to improved detection of frame-level manipulation artifacts.
• Extensive ablation studies confirm that all components—FM, LGL, TGL, and SFF—positively contribute to the final detection performance.
• The model shows strong robustness against external perturbations such as lighting changes and noise, indicating improved resilience compared to baseline models.