[Paper Review] Use of a Capsule Network to Detect Fake Images and Videos

Huy H. Nguyen, Junichi Yamagishi+1 more|arXiv (Cornell University)|Oct 28, 2019

Digital Media Forensic DetectionComputer Science69 references132 citations

TL;DR

The paper proposes Capsule-Forensics, a capsule-network-based detector that generalizes across various fake-image/video attacks, achieving competitive accuracy with far fewer parameters than CNN baselines and providing analysis through visualization of capsule activations.

ABSTRACT

The revolution in computer hardware, especially in graphics processing units and tensor processing units, has enabled significant advances in computer graphics and artificial intelligence algorithms. In addition to their many beneficial applications in daily life and business, computer-generated/manipulated images and videos can be used for malicious purposes that violate security systems, privacy, and social trust. The deepfake phenomenon and its variations enable a normal user to use his or her personal computer to easily create fake videos of anybody from a short real online video. Several countermeasures have been introduced to deal with attacks using such videos. However, most of them are targeted at certain domains and are ineffective when applied to other domains or new attacks. In this paper, we introduce a capsule network that can detect various kinds of attacks, from presentation attacks using printed images and replayed videos to attacks using fake videos created using deep learning. It uses many fewer parameters than traditional convolutional neural networks with similar performance. Moreover, we explain, for the first time ever in the literature, the theory behind the application of capsule networks to the forensics problem through detailed analysis and visualization.

Motivation & Objective

• Address the need for a general, lightweight detector for diverse manipulations of images and videos (including deepfakes, face reenactment, and CGI) that transfers across attack types.
• Leverage capsule networks to preserve hierarchical features and improve forensics task performance with fewer parameters than traditional CNNs.
• Provide theoretical and visual analysis of how capsule networks operate on forensic inputs to justify their applicability to forensics problems.

Proposed method

• Preprocess inputs by patching images or extracting frames from videos; optionally crop facial regions for face-focused detection.
• Use a VGG-19 based feature extractor up to the third max-pooling layer as a regularized front-end before the capsule network.
• Implement a Capsule-Forensics architecture with multiple primary capsules (3 or 10), each consisting of a 2D conv, a statistical pooling layer, and a 1D conv, feeding into two output capsules (real and fake).
• Apply dynamic routing with two regularizations (random noise in routing matrices and dropout) during training, plus a squash activation to stabilize learning.
• Train with cross-entropy loss and the Adam optimizer; aggregate frame/patch scores (and average over frames for video) to produce final decisions.

Experimental results

Research questions

• RQ1 Can Capsule-Forensics detect a range of manipulated content (CGI, printed/replayed attacks, and deepfake/reenactment videos) in a single framework?
• RQ2 Does using capsule networks with regularizations and larger inputs improve cross-attack detection while reducing parameter count compared to CNN baselines?
• RQ3 How do the learned capsules correspond to manipulated regions, and what does dynamic routing reveal about agreement among capsules over time?
• RQ4 Is multi-class capability feasible with capsule networks for distinguishing among specific manipulation types (Deepfakes, Face2Face, FaceSwap) beyond binary real/fake?

Key findings

• Capsule-Forensics with 300x300 inputs and enhanced settings achieves competitive binary accuracy (≈91.65%) and robust EER (≈11.36%) while using significantly fewer parameters than XceptionNet (≈3.9M vs. ≈20.8M).
• Adding random noise and dropout regularizations during routing improves performance, particularly with larger input sizes and more primary capsules.
• Increasing the number of primary capsules to 10 and applying dropout/noise yields strong multi-class performance, with Capsule-Forensics + Dropout + Noise (300x300) reaching ≈91.22% multi-class accuracy and ≈10.26% EER.
• Video frame aggregation further improves binary and multi-class accuracy, e.g., Capsule-Forensics + Dropout + Noise (video) achieving binary accuracy 93.11% and multi-class accuracy 92.90%.
• Compared to XceptionNet, Capsule-Forensics with optimizations approaches similar binary accuracy while using about five times fewer parameters; it also shows more balanced performance across manipulation types in multi-class settings.
• For fully CGI vs. photographic image discrimination on large-scale data, both old and new Capsule-Forensics variants outperform baselines, achieving 100% accuracy on the large-scale CGI/PI dataset in the reported setup.