[Paper Review] Capsule-Forensics: Using Capsule Networks to Detect Forged Images and Videos

Huy H. Nguyen, Junichi Yamagishi+1 more|arXiv (Cornell University)|Oct 26, 2018

Digital Media Forensic Detection48 citations

TL;DR

Capsule-Forensics uses a capsule-network architecture to detect a wide range of forged images and videos, including replay attacks and computer-generated forgeries, with state-of-the-art or near-perfect accuracy on several benchmarks.

ABSTRACT

Recent advances in media generation techniques have made it easier for attackers to create forged images and videos. State-of-the-art methods enable the real-time creation of a forged version of a single video obtained from a social network. Although numerous methods have been developed for detecting forged images and videos, they are generally targeted at certain domains and quickly become obsolete as new kinds of attacks appear. The method introduced in this paper uses a capsule network to detect various kinds of spoofs, from replay attacks using printed images or recorded videos to computer-generated videos using deep convolutional neural networks. It extends the application of capsule networks beyond their original intention to the solving of inverse graphics problems.

Motivation & Objective

• Motivate the need for a general, attack-agnostic forgery detector that remains effective as attack types evolve.
• Develop a capsule-network based detector that can handle both images and videos (frame-level and video-level aggregation).
• Leverage inverse-graphics-inspired capsules to capture hierarchical pose relationships and part-whole consistency for forgery cues.

Proposed method

• Extract latent features with a portion of VGG-19 from face-aligned 128x128 inputs.
• Pass features to a three-primary, two-output capsule network (real vs fake).
• Apply dynamic routing with random Gaussian noise to improve generalization and stability.
• Use a cross-entropy loss with a multi-dimensional output capsule formulation to separate real/fake along each dimension.
• Aggregate frame-level posteriors for video-level decision in videos.
• Experiment with and without added training-time random noise (Capsule-Forensics vs Capsule-Forensics-Noise).

Experimental results

Research questions

• RQ1 Can capsule networks robustly detect a broad spectrum of forgery types (replay, CGI, reenactment) beyond domain-specific detectors?
• RQ2 Does introducing random noise during training improve generalization and performance across datasets?
• RQ3 How does Capsule-Forensics compare to state-of-the-art methods on replay, face-swapping, facial reenactment, and CGI detection tasks?

Key findings

• On Replay-Attack, Capsule-Forensics achieves an HTER of 0.28% without noise and 0.00% with noise.
• On frame-level face-swapping detection (deepfake dataset), Capsule-Forensics reaches 94.47% accuracy without noise and 95.93% with noise.
• On video-level face-swapping detection, Capsule-Forensics achieves 97.69% without noise and 99.23% with noise.
• On facial reenactment (FaceForensics), Capsule-Forensics matches or exceeds the best methods at frame level across compression levels; at video level, Capsule-Forensics (with noise) achieves 99.33% vs competitors around 82–98% depending on compression.
• On CGIs vs PIs, Capsule-Forensics attains 99.72% full-size accuracy (96.75% patch) and Capsule-Forensics-Noise attains 100.00% full-size accuracy.
• Random-noise training generally improves performance across tasks and datasets.