[Paper Review] On the Detection of Digital Face Manipulation

Hao Dang, Feng Liu+3 more|arXiv (Cornell University)|Oct 3, 2019

Digital Media Forensic DetectionComputer Science61 references41 citations

TL;DR

The paper introduces an attention-based layer to improve detection of manipulated face images and localizes manipulated regions, supported by a new Diverse Fake Face Dataset (DFFD).

ABSTRACT

Detecting manipulated facial images and videos is an increasingly important topic in digital media forensics. As advanced face synthesis and manipulation methods are made available, new types of fake face representations are being created which have raised significant concerns for their use in social media. Hence, it is crucial to detect manipulated face images and localize manipulated regions. Instead of simply using multi-task learning to simultaneously detect manipulated images and predict the manipulated mask (regions), we propose to utilize an attention mechanism to process and improve the feature maps for the classification task. The learned attention maps highlight the informative regions to further improve the binary classification (genuine face v. fake face), and also visualize the manipulated regions. To enable our study of manipulated face detection and localization, we collect a large-scale database that contains numerous types of facial forgeries. With this dataset, we perform a thorough analysis of data-driven fake face detection. We show that the use of an attention mechanism improves facial forgery detection and manipulated region localization.

Motivation & Objective

• Address the rising need to detect and localize digital face manipulations across multiple manipulation types.
• Develop an attention-based mechanism that highlights informative regions and improves binary detection of genuine vs manipulated faces.
• Create a large, diverse dataset (DFFD) with real and manipulated faces to enable robust data-driven analysis.
• Evaluate the attention mechanism’s impact on detection accuracy and localization quality across seen and unseen manipulation methods.

Proposed method

• Insert an attention-based layer into a CNN classifier to produce an attention map (M_att) that highlights manipulated regions.
• Implement two attention map generation approaches: Manipulation Appearance Model (MAM) and direct regression, using either PCA-based bases or end-to-end conv nets.
• Train with a combined loss L = L_classifier + lambda * L_map, where L_map can be supervised, weakly supervised, or unsupervised.
• Use supervised L_map with pixel-level ground-truth masks, weak supervision with partial or uncertain masks, or unsupervised learning when only image labels are available.
• Evaluate detection using EER, AUC, TDR at low FDR, and localization using PBCA, IoU, Cosine similarity, and the novel IINC metric.

Experimental results

Research questions

• RQ1 Can an attention mechanism improve both detection accuracy and localization of manipulated facial regions across diverse manipulation types?
• RQ2 How do supervised, weakly supervised, and unsupervised training of the attention map affect detection and localization performance?
• RQ3 Does the proposed attention approach generalize to unseen manipulation types and external datasets?
• RQ4 What is the impact of different backbone networks (e.g., XceptionNet, VGG16) on the effectiveness of the attention layer?

Key findings

• Attention-enhanced models outperform baselines, especially at low false-detection rates (e.g., TDR 0.01%).
• Direct regression for the attention map yields best low-FDR detection, while MAM offers advantages under weak supervision or unsupervised settings.
• The attention mechanism consistently improves detection across fake types like identity and expression swap and attribute manipulation.
• The approach achieves state-of-the-art results on Celeb-DF and competitive results on UADFV when trained on the Diverse Fake Face Dataset (DFFD).
• The authors introduce IINC, a robust metric for evaluating localization that addresses limitations of IoU and Cosine Similarity on manipulated-face maps.