[Paper Review] Multi-task Learning For Detecting and Segmenting Manipulated Facial Images and Videos

Huy H. Nguyen, Fuming Fang+2 more|arXiv (Cornell University)|Jun 17, 2019

Digital Media Forensic DetectionComputer Science31 references40 citations

TL;DR

The paper introduces a Y-shaped autoencoder that jointly detects manipulated facial images/videos and segments manipulated regions, using semi-supervised learning to improve both tasks and generalize to unseen attacks.

ABSTRACT

Detecting manipulated images and videos is an important topic in digital media forensics. Most detection methods use binary classification to determine the probability of a query being manipulated. Another important topic is locating manipulated regions (i.e., performing segmentation), which are mostly created by three commonly used attacks: removal, copy-move, and splicing. We have designed a convolutional neural network that uses the multi-task learning approach to simultaneously detect manipulated images and videos and locate the manipulated regions for each query. Information gained by performing one task is shared with the other task and thereby enhance the performance of both tasks. A semi-supervised learning approach is used to improve the network's generability. The network includes an encoder and a Y-shaped decoder. Activation of the encoded features is used for the binary classification. The output of one branch of the decoder is used for segmenting the manipulated regions while that of the other branch is used for reconstructing the input, which helps improve overall performance. Experiments using the FaceForensics and FaceForensics++ databases demonstrated the network's effectiveness against facial reenactment attacks and face swapping attacks as well as its ability to deal with the mismatch condition for previously seen attacks. Moreover, fine-tuning using just a small amount of data enables the network to deal with unseen attacks.

Motivation & Objective

• Motivate robust detection of manipulated facial content in images and videos.
• Develop a system that simultaneously classifies authenticity and localizes manipulated regions.
• Explore sharing information across tasks to improve both classification and segmentation performance.
• Leverage semi-supervised learning to enhance generalization to unseen attacks.

Proposed method

• Propose a convolutional neural network with an encoder and a Y-shaped decoder for joint detection and segmentation.
• Use an activation-based latent space partitioning to route information to the appropriate decoder branch.
• Train with three losses: activation loss, segmentation loss, and reconstruction loss, combined with equal weights.
• Apply a semi-supervised training regime to improve generalization.
• Evaluate on FaceForensics and FaceForensics++ datasets, including match/mismatch and unseen-attack scenarios.
• Fine-tune with small samples to adapt to unseen attacks.

Experimental results

Research questions

• RQ1 Can a multi-task autoencoder jointly detect manipulation and localize manipulated regions in facial content?
• RQ2 Does sharing information among classification, segmentation, and reconstruction tasks improve performance over single-task baselines?
• RQ3 How well does the model generalize to unseen attacks and under varying compression levels?
• RQ4 Can small amounts of fine-tuning adapt the model to new manipulation methods?

Key findings

• Deeper networks significantly improve classification accuracy over shallower baselines (e.g., Deeper_FT achieves 93.63% accuracy in Test 1).
• The proposed New setting with equal task weights yields strong segmentation accuracy (e.g., 90.27% in Test 1) and competitive classification performance.
• Reconstruction branch and residual-input variants enhance robustness to mismatch conditions and aid segmentation.
• Unseen attacks reduce accuracy substantially across all methods, but segmentation remains relatively informative (e.g., Test 4 shows segmentation can remain meaningful).
• Fine-tuning with a small amount of data (e.g., 10 frames per video) substantially improves both classification and segmentation—FT_Res, No_Recon, and Proposed_New show notable gains.
• The proposed approach demonstrates faster adaptation to unseen attacks compared with some baselines and supports extension to audiovisual domains.