[Paper Review] Learning Rich Features for Image Manipulation Detection

Peng Zhou, Xintong Han+2 more|arXiv (Cornell University)|May 13, 2018

Digital Media Forensic DetectionComputer Science24 references82 citations

TL;DR

A two-stream Faster R-CNN detects tampered regions by combining RGB tampering artifacts with SRM-based noise features, achieving state-of-the-art results on multiple datasets and robustness to resizing/compression.

ABSTRACT

Image manipulation detection is different from traditional semantic object detection because it pays more attention to tampering artifacts than to image content, which suggests that richer features need to be learned. We propose a two-stream Faster R-CNN network and train it endto- end to detect the tampered regions given a manipulated image. One of the two streams is an RGB stream whose purpose is to extract features from the RGB image input to find tampering artifacts like strong contrast difference, unnatural tampered boundaries, and so on. The other is a noise stream that leverages the noise features extracted from a steganalysis rich model filter layer to discover the noise inconsistency between authentic and tampered regions. We then fuse features from the two streams through a bilinear pooling layer to further incorporate spatial co-occurrence of these two modalities. Experiments on four standard image manipulation datasets demonstrate that our two-stream framework outperforms each individual stream, and also achieves state-of-the-art performance compared to alternative methods with robustness to resizing and compression.

Motivation & Objective

• Motivate the need for rich feature learning to detect tampering artifacts rather than relying solely on image content.
• Propose a two-stream architecture integrating RGB visual cues and noise-based features for tamper localization.
• Train end-to-end to localize tampered regions and classify tampering types.
• Demonstrate robustness to common post-processing like resizing and JPEG compression.

Proposed method

• Two-stream Faster R-CNN with an RGB stream and a noise stream based on SRM filters.
• RPN proposals are generated from RGB features to locate likely manipulated regions.
• Noise stream processes RGB input through an SRM filter layer to extract local noise features.
• Bilinear pooling fRGB^T fN fuses RoI features from both streams for tampering classification.
• Compact bilinear pooling is used to reduce memory while preserving feature interactions.
• Loss combines RPN loss, tampering classification loss, and bounding-box regression loss.

Experimental results

Research questions

• RQ1 Can a two-stream architecture leveraging both RGB tampering artifacts and local noise inconsistencies outperform single-stream approaches in image manipulation detection?
• RQ2 What is the benefit of fusing RGB and noise features via bilinear pooling for both localization and tampering classification?
• RQ3 How robust is the proposed method to common post-processing such as resizing and JPEG compression?
• RQ4 Can the model distinguish between different tampering techniques (splicing, removal, copy-move) across datasets?

Key findings

• The two-stream RGB-N network outperforms each individual stream on four standard datasets.
• RGB features are more suitable for RPN proposal generation than noise features in this setup.
• Fusion via bilinear pooling improves tampering classification and localization beyond late fusion baselines.
• On synthetic pretraining, RGB-N achieved AP 0.627 versus 0.445 (RGB-only) and 0.461 (noise-only) in a COCO-based pretraining setup.
• Across datasets, RGB-N yields higher pixel-level F1 and AUC than several baselines, with notable gains on NIST16, Columbia, COVER, and CASIA datasets.
• The method shows robustness to JPEG quality changes and resizing attacks compared to baselines.