[Paper Review] Generate, Segment and Refine: Towards Generic Manipulation Segmentation

Peng Zhou, Bor-Chun Chen+5 more|arXiv (Cornell University)|Nov 24, 2018

Digital Media Forensic Detection41 references11 citations

TL;DR

This paper proposes GSR-Net, a novel framework for generic manipulation segmentation that generates realistic tampered images using a GAN-based generator to address data scarcity, and employs a boundary-aware segmentation and refinement pipeline to improve detection of manipulation artifacts. The method achieves state-of-the-art performance across four benchmark datasets by focusing on boundary-level artifacts rather than semantic content.

ABSTRACT

Detecting manipulated images has become a significant emerging challenge. The advent of image sharing platforms and the easy availability of advanced photo editing software have resulted in a large quantities of manipulated images being shared on the internet. While the intent behind such manipulations varies widely, concerns on the spread of fake news and misinformation is growing. Current state of the art methods for detecting these manipulated images suffers from the lack of training data due to the laborious labeling process. We address this problem in this paper, for which we introduce a manipulated image generation process that creates true positives using currently available datasets. Drawing from traditional work on image blending, we propose a novel generator for creating such examples. In addition, we also propose to further create examples that force the algorithm to focus on boundary artifacts during training. Strong experimental results validate our proposal.

Motivation & Objective

• Address the lack of comprehensive training data for manipulated image segmentation due to labor-intensive labeling.
• Improve generalization by generating hard, realistic tampered image examples that mimic real-world manipulations.
• Focus the model’s attention on boundary artifacts—key indicators of manipulation—rather than semantic content.
• Develop a self-refining training loop that enhances detection of subtle manipulation clues through iterative refinement.
• Enable effective segmentation using only COCO-style object annotations, reducing dependency on specialized annotated datasets.

Proposed method

• Propose a GAN-based generator that optimizes for realism by blending tampered regions into source images using a blending-based objective function.
• Design a three-stage GSR-Net architecture: (1) generation of hard examples from copy-pasted images, (2) boundary-guided semantic segmentation using feature concatenation, and (3) refinement by replacing predicted boundaries with authentic regions to create new training samples.
• Integrate a boundary branch that detects manipulation artifacts and a segmentation branch that fills in the interior based on boundary predictions.
• Use the refinement stage to generate new, realistic tampered images with novel boundary artifacts, which are then fed back into the segmentation network for continued learning.
• Leverage lower-level features in the segmentation network to enhance boundary localization and improve feature representation.
• Train the entire framework end-to-end with shared weights in the refinement stage for efficient inference.

Experimental results

Research questions

• RQ1 Can a GAN-based generator produce realistic, hard-to-detect manipulated image examples that improve segmentation generalization?
• RQ2 Does focusing on boundary artifacts rather than semantic content lead to better detection of image manipulations?
• RQ3 Can a self-refining training loop that iteratively generates new examples with novel boundary artifacts improve model robustness?
• RQ4 To what extent can a model trained only on COCO-style object annotations generalize to manipulation segmentation without relying on specialized annotated datasets?
• RQ5 How does the proposed framework compare to existing methods in terms of robustness under JPEG compression and image scaling attacks?

Key findings

• GSR-Net achieves state-of-the-art F1 scores on four benchmark datasets: 0.418 on Carvalho, 0.479 on In-The-Wild, 0.381 on COVER, and 0.331 on CASIA, outperforming prior methods.
• The refinement stage (CP + GSR) improves F1 score by 6.5% on Carvalho and 9.3% on In-The-Wild compared to the baseline (CP + S), demonstrating its effectiveness in focusing on boundary artifacts.
• Using generated images (DL + G) outperforms using only copy-pasted images (DL + CP), and combining both (DL + CP + G) yields the best performance, showing complementary benefits.
• The model maintains stable performance under JPEG compression and image scaling attacks, outperforming RGB-N, EXIF-selfconsistency, and MFCN in robustness.
• Even when trained only on COCO object annotations and without using CASIA 2.0’s manipulated data, GSR-Net achieves an F1 score of 0.331 on CASIA, indicating strong generalization.
• Qualitative results show that the boundary branch successfully detects subtle manipulation artifacts, and the segmentation branch accurately fills in manipulated regions across both splicing and copy-move manipulations.