[Paper Review] Two-Stream Neural Networks for Tampered Face Detection

Peng Zhou, Xintong Han+2 more|arXiv (Cornell University)|Mar 29, 2018

Digital Media Forensic DetectionComputer Science15 references57 citations

TL;DR

The paper proposes a two-stream network for tampered face detection: a GoogLeNet-based face classification stream and a patch-based triplet stream leveraging steganalysis features, evaluated on a new SwapMe/FaceSwap dataset, achieving state-of-the-art results.

ABSTRACT

We propose a two-stream network for face tampering detection. We train GoogLeNet to detect tampering artifacts in a face classification stream, and train a patch based triplet network to leverage features capturing local noise residuals and camera characteristics as a second stream. In addition, we use two different online face swapping applications to create a new dataset that consists of 2010 tampered images, each of which contains a tampered face. We evaluate the proposed two-stream network on our newly collected dataset. Experimental results demonstrate the effectiveness of our method.

Motivation & Objective

• Motivate robust detection of tampered faces beyond single-evidence methods.
• Combine high-level visual tampering artifacts with low-level local noise residual features.
• Develop and evaluate a two-stream architecture on a challenging face tampering dataset.

Proposed method

• Two-stream architecture combining: (1) a face classification stream based on GoogLeNet trained to detect tampering artifacts, and (2) a patch-based triplet stream using steganalysis features refined with a triplet loss.
• Triplet network trained on 128x128 patches with 15000 triplets; uses a 2-layer fully connected embedder with 512-d output and L2 normalization; triplet loss enforces d(a,f(r(xa))) near d(a,f(r(xp))) plus margin m.
• Per-image SVM classifier trained on learned triplet embeddings to localize tampered patches; final face tampering score combines both streams via score fusion.
• Dataset creation using two face-swapping apps (SwapMe and FaceSwap) to produce 2010 tampered images and 1400 authentic images per subset for training/testing; evaluation uses ROC/AUC.
• Training and testing protocols include cross-application evaluation (train on one app, test on the other) to assess robustness against different tampering techniques.

Experimental results

Research questions

• RQ1 Can a two-stream network improve tampered face detection by jointly leveraging high-level tampering artifacts and low-level steganalysis features?
• RQ2 Does fusing the face classification stream with a patch-based triplet stream yield better detection performance than either stream alone?
• RQ3 How well does the proposed method generalize across different face-swapping tools ( SwapMe vs. FaceSwap) under cross-dataset evaluation?

Key findings

• Two-stream fusion achieves higher AUC than either stream alone (0.927 vs. 0.854 and 0.875 on respective streams).
• Steganalysis features with SVM reach 0.794 AUC, while the face classification stream reaches 0.854 AUC.
• The combined two-stream network outperforms all baselines on the SwapMe test set with an AUC of 0.927.
• Hard JPEG-based localization methods perform worse on the SwapMe data due to double JPEG compression and resizing effects; the proposed approach remains robust.
• CAM visualizations show the face classifier learns artifacts like boundary stitching, sharp edges near lips, and blur around glasses when tampering occurs, validating learned cues.
• The approach benefits from complementary cues: visual tampering artifacts and local noise residuals are both informative for tampered-face detection.