[Paper Review] Adversarial Scene Editing: Automatic Object Removal from Weak Supervision

Rakshith Shetty, Mario Fritz+1 more|arXiv (Cornell University)|Jun 5, 2018

Generative Adversarial Networks and Image SynthesisComputer Science51 citations

TL;DR

A two-stage, interaction-free object removal model for general scene images trained with weak supervision, using mask generation and in-painting within a GAN framework with shape priors to remove objects without ground-truth targets.

ABSTRACT

While great progress has been made recently in automatic image manipulation, it has been limited to object centric images like faces or structured scene datasets. In this work, we take a step towards general scene-level image editing by developing an automatic interaction-free object removal model. Our model learns to find and remove objects from general scene images using image-level labels and unpaired data in a generative adversarial network (GAN) framework. We achieve this with two key contributions: a two-stage editor architecture consisting of a mask generator and image in-painter that co-operate to remove objects, and a novel GAN based prior for the mask generator that allows us to flexibly incorporate knowledge about object shapes. We experimentally show on two datasets that our method effectively removes a wide variety of objects using weak supervision only

Motivation & Objective

• Enable automatic object removal in general scene images without bounding boxes or masks.
• Leverage image-level labels and unpaired data to learn removal via GANs.
• Prevent degenerate solutions by coupling a mask-generator with an in-painter in a two-stage architecture.
• Impose flexible mask priors (rectangles or unpaired segmentation masks) via Wasserstein GANs to encourage coherent masks.
• Demonstrate removal performance on COCO and logo datasets with results comparable to fully supervised baselines.

Proposed method

• Two-stage editor with a mask generator G_M and an image in-painter G_I that collaborate to remove objects.
• The mask generator is trained to fool a target object classifier while the in-painter learns to fill masked regions to produce realistic output.
• A GAN-based prior enforces mask shapes through a prior discriminator D_M and a prior loss L_prior.
• In-painting is trained with reconstruction losses on random patches, a local real/fake loss, and image quality losses (tv, style) to produce coherent textures.
• Alternate optimization of G_M and G_I to prevent degenerate solutions and allow co-adaptation.
• Reconstruction and perceptual losses guide the in-painter to maintain image fidelity while removing targeted objects.

Experimental results

Research questions

• RQ1 Can general scene-level object removal be learned from weak supervision without ground-truth target images or precise masks?
• RQ2 Does a two-stage editor (mask generation + in-painting) improve removal quality over single-stage generative approaches in cluttered scenes?
• RQ3 How can mask priors (geometric shapes or unpaired segmentation masks) improve the coherence and accuracy of removal masks?
• RQ4 Is weakly supervised removal competitive with fully supervised methods like Mask-RCNN on standard datasets?
• RQ5 Can the framework generalize to removing non-object entities (e.g., logos) using weak supervision?

Key findings

• The two-stage editor (mask generator + in-painter) reduces degenerate solutions and achieves removal comparable to a fully supervised segmenter on COCO.
• Imposing flexible mask priors via Wasserstein distance improves mask coherence and reduces false removals while preserving image quality.
• Using unpaired segmentation masks or simple box priors leads to more accurate masks and better image quality than no-prior setups.
• The method outperforms a fully supervised baseline with GT/Mask-RCNN masks in removal under certain settings and approaches similar performance with dilated Mask-RCNN masks.
• The approach generalizes to logo removal with only image-level labels, demonstrating broader applicability beyond object removal.